Multi-mode personalized recommendation method and system of vehicle-mounted entertainment system and electronic equipment

Abstract

The invention discloses a multi-mode personalized recommendation method and system for a vehicle-mounted entertainment system and electronic equipment, and relates to the field of system recommendation, and the method comprises the steps: a modal perception data collection and preprocessing step: collecting multi-mode perception data, carrying out the preprocessing of the collected multi-mode perception data, and providing rich data support for a recommendation system; the multi-modal data fusion and feature extraction step comprises the following steps of: processing multi-modal time sequence data by using a long short-term memory network; the intelligent emotion recognition and demand inference step comprises the steps of analyzing user state information and recognizing a user emotion state based on the fusion feature vector; the personalized recommendation content generation step comprises the steps of constructing a personalized recommendation model, and inputting historical preference data, an emotional state and a vehicle driving situation of a user; the multi-mode interaction response step comprises the steps of completing interaction control and feedback of recommended contents based on an interaction mode selected by a user; the self-adaptive recommendation optimization step comprises the step of collecting feedback behaviors of a user on recommended contents in real time.

Description

Technical Field

[0001] This application relates to the field of system recommendation, and more particularly to multi-modal personalized recommendation methods for in-vehicle entertainment systems, multi-modal personalized recommendation systems for in-vehicle entertainment systems, electronic devices, storage media, and cockpit systems. Background Technology

[0002] Existing in-vehicle entertainment system recommendation technologies primarily rely on user operation history, behavior records, or voice commands for content recommendations. However, the limited data sources and model complexity result in low accuracy of recommendation results, failing to fully meet diverse user needs. Furthermore, existing interaction methods are relatively simplistic, lacking multi-dimensional information fusion, making it difficult to effectively improve the user experience.

[0003] Prior art document 1CN202410837457.1 provides a method and system for recommending in-vehicle human-machine interaction functions. The method includes: acquiring vehicle driving data in real time within a preset time period, and dividing the driving data into several corresponding driving conditions; detecting feature parameters corresponding to the driving data in real time based on the driving conditions, and calculating the correlation between the feature parameters and each driving condition in real time using a preset random forest algorithm, and determining in real time whether the correlation is greater than a preset correlation threshold; if the correlation is determined to be greater than the preset correlation threshold, training a preset backpropagation neural network using the feature parameters and the driving conditions to train a corresponding in-vehicle function recommendation model; detecting the real-time driving conditions of the vehicle during driving, and pushing in-vehicle functions corresponding to the real-time driving conditions in real time using the in-vehicle function recommendation model. This invention can quickly and accurately recommend the functions needed by the user, improving the user experience.

[0004] Prior art document 2CN202210271643.4 provides an apparatus, device, and method for in-vehicle interactive music recommendation. The recommendation method includes the following steps: recording user music playback lines to obtain user playback behavior records; recording user-vehicle interaction information to obtain interaction information records; analyzing and modeling the user music playback behavior records and the interaction information records to obtain features and a recall model; and updating the recommended music based on the features and recall model. This improves the recommended music by leveraging user feedback, thereby enhancing the user experience.

[0005] Prior art document 3CN202211627545.6 discloses a voice-interactive recommendation method, system, storage medium, and in-vehicle device. The method includes: acquiring voice control commands issued by a user, and identifying a first behavioral intention that the user expects the current vehicle to perform based on the voice control commands; processing the first behavioral intention into a behavior sequence prediction model to obtain several recommendation intentions and their model prediction probabilities; calculating the recommendation probability of each of the several recommendation intentions based on their respective model prediction probabilities and transition probabilities; the transition probability of each of the several recommendation intentions is a global transition probability or an individual transition probability; and determining a second behavioral intention for recommendation based on the recommendation probabilities of each of the several recommendation intentions.

[0006] Of the aforementioned comparative documents, while Comparison Document 1 uses data analysis based on driving conditions to recommend in-vehicle functions, its recommendation model primarily relies on random forests and backpropagation neural networks, failing to effectively consider users' immediate needs and emotional states, resulting in a lack of flexibility and personalization in the recommended functions. Comparison Document 2 recommends music based on users' playback behavior and interaction information, but this approach ignores factors such as users' emotional changes and driving safety in different driving situations, leading to a mismatch between recommended music and users' immediate needs. The voice interaction method in Comparison Document 3 relies on voice commands for intent recognition and recommendation, but this approach is limited to voice input, neglecting the use of other perceptual information (such as vision and touch), resulting in a simplistic interaction method and limited recommendation accuracy. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal personalized recommendation method, a multimodal personalized recommendation system for in-vehicle entertainment systems, an electronic device, a storage medium, and a cockpit system. It addresses at least the problems of poor personalization and contextual adaptability of recommended content, inability to respond in real-time to users' immediate emotional and driving state needs, resulting in inaccurate recommendations and a poor user experience. It also addresses the issue that multimodal interaction systems have not fully utilized various sensory data such as voice, gestures, and facial expressions, leading to limited and singular interaction methods, complex and inconvenient user operations. Furthermore, it resolves a technical problem in existing recommendation systems where user feedback mechanisms are lagging, unable to adjust recommendation strategies based on real-time user feedback, and struggling to achieve intelligent self-learning and optimization.

[0008] This invention provides the following solution:

[0009] According to a first aspect of the present invention, a multimodal personalized recommendation method for an in-vehicle entertainment system is provided, the method comprising: a step of multimodal perception data acquisition and preprocessing; a step of multimodal data fusion and feature extraction; a step of intelligent emotion recognition and demand inference; a step of personalized recommendation content generation; a step of multimodal interactive response; and a step of adaptive recommendation optimization.

[0010] The steps of modal sensing data acquisition and preprocessing include: acquiring multimodal sensing data and preprocessing the acquired multimodal sensing data separately to provide rich data support for the recommendation system;

[0011] The steps of multimodal data fusion and feature extraction include using a long short-term memory network to process multimodal time-series data;

[0012] Using multimodal data within a time window as the input vector, the LSTM's forget gate, input gate, cell state update, and output gate are used to capture the temporal change patterns of user behavior and emotions, generating a fused feature vector.

[0013] The steps of intelligent emotion recognition and demand inference include analyzing user state information based on fused feature vectors and identifying user emotional states;

[0014] Based on the user's emotional state and the vehicle's driving context, infer the user's needs;

[0015] The steps for generating personalized recommendations include building a personalized recommendation model and inputting user historical preference data, emotional state, and vehicle driving context.

[0016] By capturing the temporal changes in user preferences using LSTM and combining them with an attention mechanism to adjust the weights of each input dimension, in-vehicle entertainment recommendations that are adapted to the current context are generated.

[0017] The steps of multimodal interaction response include: based on the user's selected interaction method, completing the interaction control and feedback of recommended content;

[0018] The steps of adaptive recommendation optimization include real-time collection of user feedback on recommended content;

[0019] The feedback process of recommended content is modeled as a Markov decision process;

[0020] The recommendation strategy is updated using a deep Q-learning algorithm.

[0021] Furthermore, the modal sensing data acquisition and preprocessing steps also include,

[0022] The collection of multimodal perception data includes real-time collection of user voice commands, facial expressions, eye tracking, driving behavior data, and vehicle driving status data through a voice recognition module, facial recognition module, eye tracking module, driving behavior analysis module, and vehicle status monitoring module.

[0023] This also includes acquiring data on the vehicle's internal and external environment;

[0024] The preprocessing of the collected multimodal sensing data includes extracting Mel-frequency cepstral coefficients based on speech data;

[0025] Based on facial expression data, key facial points are extracted using a convolutional neural network.

[0026] Noise is eliminated using Kalman filtering based on driving behavior data;

[0027] It also includes the standardization of vehicle status data via the on-board bus.

[0028] Furthermore, the steps of multimodal data fusion and feature extraction also include processing multimodal time-series data using a long short-term memory network;

[0029] The multimodal data within the set time window are used as input vectors, including speech features, facial expression features, driving behavior features, and vehicle status features;

[0030] Feature extraction includes introducing an attention mechanism to dynamically adjust the weights of data from each modality;

[0031] The process involves calculating the attention score for each modality feature, standardizing it using the Softmax function to obtain the weights, and then generating a fused feature vector through weighted summation.

[0032] Furthermore, the intelligent emotion recognition and demand inference steps also include,

[0033] Analyze users' facial expressions, voice emotions, and driving behavior to identify users' emotional states;

[0034] Demand inference includes inferring users' immediate needs based on the vehicle's driving context.

[0035] Furthermore, the personalized recommendation content generation step also includes constructing a personalized recommendation model by inputting user historical preference data, emotional state, and vehicle driving context;

[0036] Input user history preference data, including music playback history and navigation habits;

[0037] Recommended in-car entertainment content tailored to the current context includes music, navigation routes, and videos.

[0038] Furthermore, the steps of multimodal interactive response also include,

[0039] The interaction methods that users can choose include voice, gesture, touch screen, and eye tracking.

[0040] By using natural language processing to recognize voice commands, computer vision to recognize gestures, optical flow to track eye movements, and / or event-driven UI responses to touch operations, interactive control and feedback for recommended content can be achieved.

[0041] Furthermore, the adaptive recommendation optimization steps also include,

[0042] Real-time collection of user feedback on recommended content includes both positive and negative feedback;

[0043] Positive feedback includes no modification, repeat playback, and increasing volume;

[0044] Negative feedback includes switching content, modifying navigation, and lowering the volume;

[0045] The feedback process of recommended content is modeled as a Markov decision process;

[0046] The recommendation strategy updated using the deep Q-learning algorithm includes setting reward values ​​based on feedback behavior, optimizing recommendation model parameters through the Q-value update formula, and dynamically adjusting subsequent recommended content until user needs are met or the user terminates the interaction.

[0047] According to a second aspect of the present invention, a multimodal personalized recommendation system for an in-vehicle entertainment system is provided, the multimodal personalized recommendation system for the in-vehicle entertainment system comprising: a module for multimodal perception data acquisition and preprocessing; a module for multimodal data fusion and feature extraction; a module for intelligent emotion recognition and demand inference; a module for personalized recommendation content generation; a module for multimodal interactive response; and a module for adaptive recommendation optimization;

[0048] The module for modal sensing data acquisition and preprocessing is used to acquire multimodal sensing data and preprocess the acquired multimodal sensing data to provide rich data support for the recommendation system.

[0049] A module for multimodal data fusion and feature extraction is used to process multimodal time-series data using long short-term memory networks;

[0050] Using multimodal data within a time window as the input vector, the LSTM's forget gate, input gate, cell state update, and output gate are used to capture the temporal change patterns of user behavior and emotions, generating a fused feature vector.

[0051] The intelligent emotion recognition and demand inference module is used to analyze user state information and identify user emotional state based on fused feature vectors.

[0052] Based on the user's emotional state and the vehicle's driving context, infer the user's needs;

[0053] The module for generating personalized recommendation content is used to build a personalized recommendation model, taking into account users' historical preference data, emotional state, and vehicle driving context.

[0054] By capturing the temporal changes in user preferences using LSTM and combining them with an attention mechanism to adjust the weights of each input dimension, in-vehicle entertainment recommendations that are adapted to the current context are generated.

[0055] The multimodal interaction response module is used to complete the interactive control and feedback of recommended content based on the user's selected interaction method;

[0056] An adaptive recommendation optimization module is used to collect user feedback on recommended content in real time;

[0057] The feedback process of recommended content is modeled as a Markov decision process;

[0058] The recommendation strategy is updated using a deep Q-learning algorithm.

[0059] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0060] The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the multi-modal personalized recommendation method of the in-vehicle entertainment system.

[0061] According to a fourth aspect of the present invention, a computer-readable storage medium is provided storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the multimodal personalized recommendation method of the in-vehicle entertainment system.

[0062] According to a fifth aspect of the present invention, a cockpit system is provided, comprising:

[0063] An electronic device for implementing the multi-modal personalized recommendation method of the in-vehicle entertainment system;

[0064] The processor runs a program that, when running, executes the steps of the multimodal personalized recommendation method of the in-vehicle entertainment system from data output by the electronic device.

[0065] A storage medium for storing a program that, when running, executes the steps of the multimodal personalized recommendation method of the in-vehicle entertainment system based on data output from an electronic device.

[0066] The above solution achieves the following beneficial technical effects:

[0067] This application constructs a more accurate personalized recommendation model by collecting and processing multimodal perception data and combining users' real-time emotions, driving behavior and vehicle status.

[0068] This application makes recommendations based on users' historical preferences and can also make real-time adjustments based on the current driving context (such as road conditions, emotional state, etc.), thereby greatly improving the personalization and context adaptability of the recommended content.

[0069] This application enhances the user experience by improving personalized and contextualized recommendations, making the recommended entertainment content more aligned with the user's immediate needs. Attached Figure Description

[0070] Figure 1 This is a flowchart of a multi-modal personalized recommendation method for an in-vehicle entertainment system provided by one or more embodiments of the present invention.

[0071] Figure 2 This is a structural diagram of a multi-modal personalized recommendation system for an in-vehicle entertainment system provided by one or more embodiments of the present invention.

[0072] Figure 3 This is a schematic diagram of the overall system architecture provided in a specific embodiment of the present invention.

[0073] Figure 4 This is a schematic diagram of a multimodal sensing data acquisition process provided in a specific embodiment of the present invention.

[0074] Figure 5 This is a schematic diagram of the workflow of a personalized recommendation model provided in a specific embodiment of the present invention.

[0075] Figure 6 This is a schematic diagram of a reinforcement learning feedback optimization process provided in a specific embodiment of the present invention.

[0076] Figure 7 This is a schematic diagram illustrating multimodal interaction provided in a specific embodiment of the present invention.

[0077] Figure 8 This is a schematic diagram of a data flow provided in a specific embodiment of the present invention.

[0078] Figure 9 This is an electronic device structural block diagram of a multi-modal personalized recommendation method for an in-vehicle entertainment system provided by one or more embodiments of the present invention. Detailed Implementation

[0079] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0080] Figure 1 This is a flowchart of a multi-modal personalized recommendation method for an in-vehicle entertainment system provided by one or more embodiments of the present invention.

[0081] like Figure 1 The multimodal personalized recommendation method of the in-vehicle entertainment system shown includes: step S1 of multimodal perception data acquisition and preprocessing; step S2 of multimodal data fusion and feature extraction; step S3 of intelligent emotion recognition and demand inference; step S4 of personalized recommendation content generation; step S5 of multimodal interaction response; and step S6 of adaptive recommendation optimization.

[0082] Step S1 of modal perception data acquisition and preprocessing includes acquiring multimodal perception data and preprocessing the acquired multimodal perception data respectively to provide rich data support for the recommendation system;

[0083] Step S2 of multimodal data fusion and feature extraction includes processing multimodal time-series data using a long short-term memory network;

[0084] Using multimodal data within a time window as the input vector, the LSTM's forget gate, input gate, cell state update, and output gate are used to capture the temporal change patterns of user behavior and emotions, generating a fused feature vector.

[0085] Step S3 of intelligent emotion recognition and demand inference includes analyzing user state information and identifying user emotional state based on fused feature vectors;

[0086] Based on the user's emotional state and the vehicle's driving context, infer the user's needs;

[0087] Step S4 of generating personalized recommendation content includes: building a personalized recommendation model and inputting user historical preference data, emotional state and vehicle driving context;

[0088] By capturing the temporal changes in user preferences using LSTM and combining them with an attention mechanism to adjust the weights of each input dimension, in-vehicle entertainment recommendations that are adapted to the current context are generated.

[0089] Step S5 of the multimodal interaction response includes completing the interactive control and feedback of the recommended content based on the user's selected interaction method;

[0090] Step S6 of adaptive recommendation optimization includes real-time collection of user feedback behavior on recommended content;

[0091] The feedback process of recommended content is modeled as a Markov decision process;

[0092] The recommendation strategy is updated using a deep Q-learning algorithm.

[0093] Specifically, the purpose of this application is to provide a strategy for intelligent recommendation and interaction in in-vehicle entertainment systems. By integrating multiple data sources, the intelligence level and interactive experience of the recommendation system are improved, overcoming the problems of low recommendation accuracy and single interaction methods in the prior art.

[0094] In this embodiment, the modal sensing data acquisition and preprocessing steps further include,

[0095] The collection of multimodal perception data includes real-time collection of user voice commands, facial expressions, eye tracking, driving behavior data, and vehicle driving status data through a voice recognition module, facial recognition module, eye tracking module, driving behavior analysis module, and vehicle status monitoring module.

[0096] This also includes acquiring data on the vehicle's internal and external environment;

[0097] The preprocessing of the collected multimodal sensing data includes extracting Mel-frequency cepstral coefficients based on speech data;

[0098] Based on facial expression data, key facial points are extracted using a convolutional neural network.

[0099] Noise is eliminated using Kalman filtering based on driving behavior data;

[0100] It also includes the standardization of vehicle status data via the on-board bus.

[0101] Specifically, the system uses a voice recognition module, a facial recognition module, an eye-tracking module, a driving behavior analysis module, and a vehicle status monitoring module to collect user voice commands, facial expressions, eye movements, driving behavior data (steering wheel rotation rate, acceleration, braking force), and vehicle driving status data (speed, fuel consumption, driving mode) in real time. It also acquires internal and external environmental data (weather, road conditions, time). The collected multimodal data undergoes preprocessing: Mel-frequency cepstral coefficients (MFCC) are extracted from voice data; facial expression data is processed using a convolutional neural network (CNN) to extract facial key points; driving behavior data undergoes Kalman filtering to eliminate noise; and vehicle status data is standardized using the vehicle bus.

[0102] The voice recognition module collects voice data through the vehicle microphone, the face recognition module and the eye tracking module share the vehicle camera, the driving behavior analysis module collects data through the gyroscope and accelerometer, and the vehicle status monitoring module obtains vehicle driving status data through the CAN bus.

[0103] In this embodiment, the steps of multimodal data fusion and feature extraction further include processing multimodal time-series data using a long short-term memory network;

[0104] The multimodal data within the set time window are used as input vectors, including speech features, facial expression features, driving behavior features, and vehicle status features;

[0105] Feature extraction includes introducing an attention mechanism to dynamically adjust the weights of data from each modality;

[0106] The process involves calculating the attention score for each modality feature, standardizing it using the Softmax function to obtain the weights, and then generating a fused feature vector through weighted summation.

[0107] Specifically, a Long Short-Term Memory (LSTM) network is used to process multimodal temporal data. The multimodal data within a time window is set as the input vector (including speech features, facial expression features, driving behavior features, and vehicle state features). The LSTM's forget gate, input gate, cell state update, and output gate are used to capture the temporal change patterns of user behavior and emotions. An attention mechanism is introduced to dynamically adjust the weights of each modality: the attention score of each modality feature is calculated, and the weights are obtained after standardization by the Softmax function. Then, a fused feature vector is generated by weighted summation.

[0108] In this embodiment, the steps of intelligent emotion recognition and demand inference further include,

[0109] Analyze users' facial expressions, voice emotions, and driving behavior to identify users' emotional states;

[0110] Demand inference includes inferring users' immediate needs based on the vehicle's driving context.

[0111] Specifically, it analyzes users' facial expressions (such as the angle of eyebrow raising and the curvature of the corners of the mouth), voice emotions (tone and speed of speech) and driving behavior (frequency of rapid acceleration / braking) to identify users' emotional states (tension, relaxation, fatigue, anxiety); combined with the vehicle driving context (high-speed driving, urban congestion), it infers users' immediate needs (such as relaxation and entertainment, efficient navigation).

[0112] The specific judgment logic of "intelligent emotion recognition" includes: if the vehicle is traveling at high speed and the user's facial expression is smiling, the tone of voice is calm, and the driving behavior is without sudden acceleration / sudden braking, then it is judged as a "relaxed" state; if the vehicle is in urban congestion and the user's facial expression is frowning, the tone of voice is rapid, and the driving behavior is frequent sudden braking, then it is judged as an "anxious" state.

[0113] In this embodiment, the step of generating personalized recommendation content further includes constructing a personalized recommendation model by inputting user historical preference data, emotional state, and vehicle driving context.

[0114] Input user history preference data, including music playback history and navigation habits;

[0115] Recommended in-car entertainment content tailored to the current context includes music, navigation routes, and videos.

[0116] Specifically, a personalized recommendation model is constructed by inputting users' historical preference data (music playback history, navigation habits), emotional state and vehicle driving context. The model uses LSTM to capture the temporal changes in user preferences and combines an attention mechanism to adjust the weights of each input dimension to generate in-vehicle entertainment recommendation content (music, navigation route, video) that is adapted to the current context.

[0117] The contextual adjustment rules for "personalized recommended content" include: when the user is in a "relaxed" state, recommend light music with a tempo of ≤80 BPM; when the user is in an "anxious" state, recommend the optimal navigation route and reduce unnecessary notification pushes; when the vehicle speed is ≥80km / h, recommend voice interaction as the preferred operation mode and disable complex touch screen operations.

[0118] In this embodiment, the multimodal interactive response step further includes,

[0119] The interaction methods that users can choose include voice, gesture, touch screen, and eye tracking.

[0120] By using natural language processing to recognize voice commands, computer vision to recognize gestures, optical flow to track eye movements, and / or event-driven UI responses to touch operations, interactive control and feedback for recommended content can be achieved.

[0121] Specifically, based on the user's selected interaction method (voice, gesture, touch screen, eye tracking), natural language processing (NLP) is used to recognize voice commands, computer vision (CV) is used to recognize gestures, optical flow is used to track eye movements, and event-driven UI responses to touch operations are used to complete the interactive control and feedback of recommended content.

[0122] The technical details of the "multimodal interactive response" include: voice interaction uses a CNN-RNN model to convert speech signals into text and recognize intent; gesture interaction uses CV technology to detect key points on the hand and classify actions (such as waving to switch music, clenching a fist to adjust the volume); eye tracking control uses optical flow to track pupil position and match the screen interaction area to complete the selection; touch screen interaction adopts event-driven UI design, with a response latency of ≤100ms for click and swipe operations.

[0123] In this embodiment, the adaptive recommendation optimization step further includes,

[0124] Real-time collection of user feedback on recommended content includes both positive and negative feedback;

[0125] Positive feedback includes no modification, repeat playback, and increasing volume;

[0126] Negative feedback includes switching content, modifying navigation, and lowering the volume;

[0127] The feedback process of recommended content is modeled as a Markov decision process;

[0128] The recommendation strategy updated using the deep Q-learning algorithm includes setting reward values ​​based on feedback behavior, optimizing recommendation model parameters through the Q-value update formula, and dynamically adjusting subsequent recommended content until user needs are met or the user terminates the interaction.

[0129] Specifically, the system collects user feedback on recommended content in real time (positive feedback: no modification, repeat playback, increase volume; negative feedback: switch content, modify navigation, decrease volume), models the recommendation process as a Markov decision process (MDP), and uses a deep Q-learning (DQN) algorithm to update the recommendation strategy: a reward value is set based on the feedback behavior, the recommendation model parameters are optimized through the Q-value update formula, and subsequent recommended content is dynamically adjusted until the user's needs are met or the user terminates the interaction.

[0130] Figure 2 This is a structural diagram of a multi-modal personalized recommendation system for an in-vehicle entertainment system provided by one or more embodiments of the present invention.

[0131] like Figure 2 The multimodal personalized recommendation system of the in-vehicle entertainment system includes: a module for multimodal perception data acquisition and preprocessing; a module for multimodal data fusion and feature extraction; a module for intelligent emotion recognition and demand inference; a module for personalized recommendation content generation; a module for multimodal interactive response; and a module for adaptive recommendation optimization.

[0132] The module for modal sensing data acquisition and preprocessing is used to acquire multimodal sensing data and preprocess the acquired multimodal sensing data to provide rich data support for the recommendation system.

[0133] A module for multimodal data fusion and feature extraction is used to process multimodal time-series data using long short-term memory networks;

[0134] Using multimodal data within a time window as the input vector, the LSTM's forget gate, input gate, cell state update, and output gate are used to capture the temporal change patterns of user behavior and emotions, generating a fused feature vector.

[0135] The intelligent emotion recognition and demand inference module is used to analyze user state information and identify user emotional state based on fused feature vectors.

[0136] Based on the user's emotional state and the vehicle's driving context, infer the user's needs;

[0137] The module for generating personalized recommendation content is used to build a personalized recommendation model, taking into account users' historical preference data, emotional state, and vehicle driving context.

[0138] By capturing the temporal changes in user preferences using LSTM and combining them with an attention mechanism to adjust the weights of each input dimension, in-vehicle entertainment recommendations that are adapted to the current context are generated.

[0139] The multimodal interaction response module is used to complete the interactive control and feedback of recommended content based on the user's selected interaction method;

[0140] An adaptive recommendation optimization module is used to collect user feedback on recommended content in real time;

[0141] The feedback process of recommended content is modeled as a Markov decision process;

[0142] The recommendation strategy is updated using a deep Q-learning algorithm.

[0143] It is worth noting that although this system / device only discloses modules for multimodal perception data acquisition and preprocessing; multimodal data fusion and feature extraction; intelligent emotion recognition and demand inference; personalized recommendation content generation; multimodal interactive response; and adaptive recommendation optimization, this does not mean that this device is limited to the above-mentioned basic functional modules. Rather, what this invention intends to express is that, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with existing technologies to form an infinite number of embodiments or technical solutions. In other words, this system / device is open rather than closed. The fact that this embodiment only discloses a few basic functional modules should not be interpreted as the scope of protection of the claims of this invention being limited to the above-disclosed basic functional modules.

[0144] In one specific embodiment, an intelligent recommendation and interaction system for an in-vehicle entertainment system is disclosed. By integrating multiple data sources, the system improves the intelligence level and interactive experience of the recommendation system, overcoming the problems of low recommendation accuracy and single interaction method in the prior art.

[0145] like Figure 3 As shown, the system includes the following components: a multimodal perception data acquisition module, an intelligent emotion recognition module, a personalized recommendation model, a multimodal interaction system, and an adaptive recommendation feedback module. Among them:

[0146] Multimodal sensing data acquisition module:

[0147] like Figure 4 As shown, the system collects various sensory data from users in real time, utilizing modules such as voice recognition, facial recognition, eye tracking, driving behavior analysis, and vehicle status monitoring to collect data including, but not limited to, user voice commands, facial expressions, eye tracking, driving behavior, and vehicle driving status. Simultaneously, it combines this with internal and external environmental data (such as weather, road conditions, and time) to provide rich data support for the recommendation system.

[0148] The system collects various sensory data in real time, including voice, facial expressions, eye tracking, driving behavior, and vehicle driving status. To effectively utilize and integrate this data, the system employs advanced machine learning algorithms such as LSTM (Long Short-Term Memory) networks and attention mechanisms. The specific steps are as follows:

[0149] 1. Speech data is collected via microphone, and the system uses acoustic models to extract features, such as Mel-frequency cepstral coefficients (MFCCs). These features represent the emotion and instructions in the speech.

[0150] 2. The camera acquires the user's facial expressions and eye-tracking data, and uses a convolutional neural network (CNN) to analyze the images and extract key facial features (such as eyes and corners of the mouth). Eye tracking is used to capture pupil position and gaze direction to determine the user's focus of attention.

[0151] 3. Driving behavior data, such as steering wheel rotation rate and acceleration, is collected by gyroscopes and accelerometers. The system uses a Kalman filter algorithm to smooth this data and eliminate noise. Vehicle driving status data, such as speed and fuel consumption, are obtained via the onboard bus.

[0152] 4. The system uses a Long Short-Term Memory (LSTM) network to process multimodal time-series data. Multimodal data within a specific time window is defined as the input vector. ,That Composed of data in different modalities, such as ,in Representing speech features, This indicates facial expression characteristics, and so on.

[0153] For the input data at each time step t The update formula for LSTM is as follows:

[0154] Forgotten Gate: ;

[0155] The forget gate determines how much state information from the previous moment is retained, among which... This is the weight matrix. This is a bias term.

[0156] Input Gate: ;

[0157] The input gate determines the weight of new information on the current state update.

[0158] Cell status update: ;

[0159] Current cell state The previous state controlled by the forget gate It is updated together with the current input controlled by the input gate.

[0160] Output gate: ;

[0161] The output gate determines the output information of the LSTM cell.

[0162] Final output: ;

[0163] In this way, the system outputs at each time step. It captures patterns in user behavior and emotions over time.

[0164] 5. Attention mechanisms for multimodal data fusion

[0165] To more effectively fuse data from different modalities, the system introduces an attention mechanism. For each time step t, the multimodal feature vector... These correspond to different data modalities (voice, facial expressions, driving behavior, etc.). The attention mechanism calculates the weights for each modality, specifically as follows:

[0166] (1) Calculate attention weights :

[0167] ;

[0168] Where W is the weight matrix and v is the attention weight vector. The weights are then standardized using the Softmax function.

[0169] ;

[0170] (2) Calculate the weighted eigenvector H:

[0171] ;

[0172] The attention mechanism enables the system to dynamically adjust the weights of different modalities, thereby making more accurate recommendations for the current driving situation.

[0173] For example, when driving on a highway, the system uses LSTM to identify that the user is exhibiting a relaxed mood and focused on the road ahead. Based on this multimodal data, the system assigns high weight to driving behavior through an attention mechanism, ultimately recommending music suitable for the situation, such as relaxing background music. This intelligent recommendation improves the user experience while reducing operational distractions during driving.

[0174] Intelligent emotion recognition module:

[0175] The system analyzes users' facial expressions, voice emotions, and driving behavior to identify their emotional state (such as tension, relaxation, fatigue, etc.) in real time. Combined with the driving context (such as highway driving, urban traffic congestion, etc.), it infers the user's immediate needs.

[0176] The core of the intelligent emotion recognition module is to analyze and judge the user's emotional state in real time through multiple information sources, such as facial expressions, voice emotions, and driving behavior, and make accurate inferences based on the driving context. The system refines the user's emotional state into several typical states, such as tension, relaxation, fatigue, and anxiety, and then infers the user's immediate needs based on these states, thereby providing suitable suggestions or function recommendations.

[0177] The system acquires facial images of the user through a camera and uses a convolutional neural network to extract key facial points, such as eyebrows, eyes, and corners of the mouth. It then analyzes facial expressions by examining the distances and trends between these key points. Specifically, the facial expression feature vector is... The vector contains multiple facial features, such as the distance between the corners of the eyes and mouth, and the angle of the eyebrows.

[0178] Facial expression classification is handled using support vector machines or neural networks. Assuming there are N emotion categories, each with a corresponding probability, the system classifies each emotion state according to the following formula. Perform the calculation:

[0179] ;

[0180] in The weights for the facial expression classification model, The current facial expression belongs to an emotional state. The probability of different emotional states is calculated using the Softmax function. This formula normalizes the probabilities of different emotional states to obtain the emotional distribution represented by the user's current expression.

[0181] The system collects users' voice data through a microphone. First, it preprocesses the voice signal to extract features such as Mel-frequency cepstral coefficients (MFCCs). These features represent the acoustic properties of the speech and reflect the user's tone, pitch, and emotional fluctuations.

[0182] The extracted features were processed using a recurrent neural network (RNN) because RNNs are well-suited for capturing the temporal dependencies of speech signals. (Settings...) For speech feature vectors, the system calculates the hidden state using RNN units. And update using the following formula:

[0183] ;

[0184] in This represents the hidden state at the current moment. and This is the weight matrix. The input is the speech feature vector. Finally, a fully connected layer maps the hidden states to sentiment classification, similar to facial expression analysis, and the probability distribution of different sentiment states is calculated using the Softmax function.

[0185] Driving behavior monitoring consists of data such as steering wheel rotation angle, acceleration, and braking force. The system uses Kalman filtering to smooth this sensor data, removing noise and extracting effective driving behavior features. These driving behavior features are defined as follows: These data are modeled using time series models such as LSTM to capture dynamic changes in behavior during driving.

[0186] Through this process, the system can detect whether the user's driving behavior is abnormal, such as frequent sudden braking or acceleration, and further determine whether the user is in a state of tension or anxiety.

[0187] The system integrates the analysis results of facial expressions, voice emotions, and driving behavior to obtain a comprehensive feature vector of the user's emotional state. The system assigns weights to different data sources using a weighted approach and dynamically adjusts these weights based on the driving context.

[0188] The comprehensive determination of emotional states is accomplished through attention mechanisms. For each type of emotional state... The system calculates the overall weight using the following formula:

[0189] ;

[0190] in It is the weight matrix for sentiment classification. Attention weights are assigned to different emotional states. The final overall emotional state is:

[0191] ;

[0192] In this way, the system can identify the user's emotional state in real time. For example, when driving on a highway, if the system detects that the user's facial expression is relaxed, their voice is steady, and their driving behavior is stable, the system judges that the user is in a relaxed state. However, if the system detects that the user brakes suddenly multiple times, their eyebrows are furrowed, and their voice shows anxiety in a congested urban environment, the system judges that the user is in a tense state.

[0193] For example, when a user is driving at high speed, the system detects a smiling facial expression through the camera, and voice recognition analysis shows that the user's tone of voice is steady and the speaking speed is moderate. Simultaneously, the driving behavior exhibits smooth acceleration and steering. In this situation, the system judges that the user is relaxed and may recommend playing relaxing background music. However, in congested urban traffic, the system detects frequent braking by analyzing driving behavior, and combined with facial expressions such as frowning and anxious tones in the voice, the system judges that the user is anxious and may provide navigation suggestions or reduce unnecessary notifications to alleviate user stress.

[0194] Personalized recommendation model:

[0195] like Figure 4As shown, the system trains a deep learning model based on multimodal data and employs algorithms such as attention mechanisms, long short-term memory networks (LSTM), and reinforcement learning to adjust recommendation strategies in real time. This model not only considers users' historical preferences but also incorporates immediate emotions and driving status to provide more personalized and contextualized entertainment content recommendations (such as music, videos, and navigation).

[0196] The input to a personalized recommendation model includes the following multimodal data:

[0197] User's historical preference data, such as music playback history and navigation usage habits, etc.

[0198] Real-time emotion data, including the results of voice, facial expressions, and driving behavior analysis.

[0199] Real-time driving status data, such as vehicle speed and current road conditions.

[0200] After preprocessing, these data form a multidimensional feature vector. ,in Indicating historical preference characteristics, Indicates the current emotional characteristics, This represents driving state characteristics. Each feature dimension extracts features through a corresponding deep learning model.

[0201] To capture the temporal dynamics of user behavior and emotions, the system uses LSTM to model time-series data. LSTM can effectively handle time-dependent input data, such as users' historical behavior and emotional changes. (Settings...) Given the input vector at time step t, the system can learn the user's emotional changes and behavioral patterns over time through the state update process of the LSTM network, and extract dynamic features related to recommendations.

[0202] To dynamically capture the impact of different modalities on recommendation decisions, the system employs an attention mechanism to adaptively adjust the weights of each modality. Specifically, for each time step t, the multimodal features... , , The attention mechanism calculates the importance weights of each feature. The formula is as follows:

[0203] Calculate the score for each feature. :

[0204] ;

[0205] Where W is the weight matrix and v is the attention weight vector.

[0206] Weights are calculated using the Softmax function. :

[0207] ;

[0208] The final weighted eigenvector H is:

[0209] ;

[0210] This mechanism allows the system to dynamically adjust the weighting of historical preferences, emotional state, and driving status in different driving scenarios. For example, when the system detects that a user is in an anxious state, it may reduce the recommendation weight of complex navigation tasks and recommend more relaxing music or soothing video content.

[0211] Multimodal interaction system:

[0212] like Figure 7 As shown, the system supports multiple interaction methods, including a voice interaction system, a gesture recognition system, a touchscreen operating system, and an eye-tracking system. Users can interact with the system through voice, gestures, touchscreen, or eye-tracking control. Utilizing natural language processing (NLP) and computer vision (CV) technologies, the system can recognize user intentions and provide corresponding responses. Users are not limited to a single interaction method but can choose the most suitable method based on the driving environment.

[0213] Voice interaction is one of the main interaction methods in multimodal systems, especially suitable for situations where manual operation is inconvenient for users while driving. The system acquires voice signals in real time via a microphone and converts them into text using end-to-end speech recognition technology. Then, a natural language processing module understands the user's intent. The system uses deep neural networks (such as CNN-RNN models) for speech recognition. Given a voice signal x, the speech recognition model outputs the corresponding text sequence y.

[0214] The system supports contactless operation via gestures, allowing users to perform common tasks such as adjusting volume, switching music, or answering calls using hand movements. Gesture recognition utilizes computer vision (CV) technology combined with deep learning to detect and classify hand movements.

[0215] Touchscreens are another common input method, suitable for when the vehicle is stationary or moving at low speeds. Users can perform precise operations via the touchscreen, such as entering navigation destinations, selecting playlists, or checking vehicle status. The system uses an event-driven UI design, enabling it to respond quickly to user touch actions.

[0216] Eye-tracking control is particularly suitable for scenarios requiring quick selection where manual or voice operation is inconvenient. The system tracks the user's gaze in real time using a camera, infers the user's focus, and then completes the operation. This module analyzes eye movement trajectories based on eye movement characteristics using optical flow or deep learning models.

[0217] Assuming a user is driving at high speed, the system achieves seamless interaction through voice and eye tracking. When the user says "play music," the system captures the voice information through the voice recognition module while simultaneously detecting that the user is looking at the music playback interface. Through multimodal fusion, the system can automatically highlight the music option in the user's line of sight as the user completes the voice command, thus responding quickly to the command.

[0218] In another scenario, with the vehicle traveling at low speed, the user inputs their navigation destination via the touchscreen. When selecting from complex menu items, they can use gestures for quick swipes, such as waving to switch map zoom levels. This combination of touchscreen and gesture interaction ensures both convenience and safety for the user.

[0219] Adaptive recommendation feedback module:

[0220] like Figure 6 As shown, the system possesses self-learning capabilities. By analyzing user feedback behavior (such as music switching and adjustments to interactive commands), it continuously optimizes the recommendation model and improves recommendation accuracy. Simultaneously, the system can detect user responses to recommended content and adjust subsequent recommendation strategies in real time, ensuring that the system can adapt to users' immediate needs.

[0221] The system monitors user feedback behaviors in real time during interactions, such as music switching, volume adjustment, and navigation route modification. These behaviors are considered positive or negative feedback from users to the recommended content. By calculating the corresponding feedback signals, the system can determine the effectiveness of the current recommended content. Feedback behaviors can be represented in the following forms:

[0222] Positive feedback: Users do not change the recommended content, or they perform positive interactive behaviors, such as increasing the number of plays or increasing the volume.

[0223] Negative feedback: Users immediately switch music, adjust recommended content, modify navigation, etc.

[0224] set up As a feedback signal, This indicates positive feedback. This indicates negative feedback. The system adjusts the recommendation model based on user feedback signals.

[0225] The adaptive recommendation feedback module optimizes the model based on the principles of reinforcement learning. The system models the recommendation process as a Markov decision process (MDP), in which the system's task is to adjust the strategy based on user feedback signals so that subsequent recommendations better match user preferences.

[0226] state The current status of the system includes information such as the user's historical preferences, current emotional state, and vehicle driving status.

[0227] action System-recommended content, such as a piece of music or a navigation route.

[0228] award User feedback signals are used by the system to assign different reward values ​​based on positive or negative feedback. For example, positive feedback... negative feedback .

[0229] Strategy The system is in state Select action The strategy.

[0230] The goal of reinforcement learning is to maximize long-term cumulative rewards. ,Right now:

[0231] ;

[0232] Here, γ is the discount factor, which determines the importance of future rewards.

[0233] The system uses the Deep Q-Learning (DQN) algorithm to optimize the recommendation strategy. The Q-value update formula is as follows:

[0234] ;

[0235] At each time step t, the system adjusts its current state accordingly. and user feedback This updates the Q-value of the recommendation model, making subsequent recommendations more accurate.

[0236] Suppose a user is driving and the system recommends relaxing music. However, the user immediately switches to a faster-paced song. At this point, the system detects a negative feedback signal. The system then uses Q-learning to lower the priority of this type of light music. In future driving environments, in similar situations, the system may recommend more fast-paced music to better suit user preferences.

[0237] In another scenario, the user adjusted the navigation route recommended by the system, indicating a deviation in the navigation recommendation. The system reduced the weight of that navigation route and adjusted the navigation algorithm through a real-time learning mechanism to make subsequent navigation recommendations more accurate.

[0238] like Figure 8 As shown, the above scheme collects multimodal data, then processes and extracts features, performs personalized recommendations through intelligent emotion recognition, and achieves iterative learning through user feedback on the recommended content.

[0239] In another screenshot example, Example 1: Multimodal Sensing Data Acquisition and Processing

[0240] Background: In-vehicle entertainment systems need to recommend suitable content based on the user's immediate needs and driving status, but accurate recommendations are difficult to achieve using only a single data source (such as the user's music history preferences). Therefore, this embodiment constructs a recommendation system capable of comprehensively perceiving the user's state and environment through the collection and processing of multimodal sensing data.

[0241] step:

[0242] Data Acquisition: The system collects various sensory data from the user in real time through multiple sensors, specifically including:

[0243] Voice data: Collects user voice commands, tone, and speaking speed through the vehicle's microphone.

[0244] Facial expressions: Captures changes in the user's facial expressions using the vehicle's camera, recognizing expressions such as smiling and frowning.

[0245] Eye tracking: Captures the user's eye movement trajectory using infrared or cameras to determine the area of ​​their attention.

[0246] Driving behavior: The vehicle control module records the user's driving behavior data, such as acceleration, braking, and steering.

[0247] Vehicle driving status: The system collects information such as the vehicle's current driving speed, fuel consumption, and driving mode from the vehicle's sensors.

[0248] Data Fusion: All collected data is processed through data fusion algorithms to generate a comprehensive description of the user's current state. For example, the system analyzes voice data and facial expressions to determine the user's emotional state (such as tension or relaxation); and combines driving behavior and vehicle status to infer the user's level of focus or fatigue.

[0249] Data Processing and Feature Extraction: The system preprocesses and extracts features from the collected data. Example:

[0250] Voice data is converted into text information through speech recognition technology, and at the same time, sentiment analysis algorithms are used to extract the user's emotional characteristics.

[0251] Facial expression data is used to extract key points through computer vision algorithms (such as convolutional neural networks CNN) to identify the types of expressions a user makes.

[0252] Driving behavior data is combined with driving patterns to extract feature values ​​(such as driving style, braking intensity, etc.).

[0253] Eye-tracking data analyzes eye movement patterns to identify a user's focus on the screen or their surroundings.

[0254] Multimodal Feature Fusion Model: After feature extraction, multimodal data is input into the fusion model. This model is based on a Long Short-Term Memory (LSTM) network and uses time series analysis to examine the correlation and trends of different modalities. It assumes a multimodal feature set. Each mode These represent different data sources. The model outputs a fused feature vector F, which is then used in subsequent recommendation models.

[0255] Implementation results: This multimodal data acquisition and processing system can perceive the user's current state and environment in real time and comprehensively, enabling the recommendation system to not only consider the user's historical data, but also dynamically adapt to the current driving situation, thereby improving the accuracy and adaptability of the recommendations.

[0256] Example 2: Adaptive Recommendation Feedback Based on Reinforcement Learning

[0257] Background: Recommendation systems often struggle to fully understand user needs in the early stages. Therefore, by using reinforcement learning mechanisms, the system can gradually learn and optimize recommended content through long-term user interactions.

[0258] step:

[0259] Initial Recommendations: During system startup, based on the user's historical preferences, current driving status, and sentiment analysis, some common content (such as music, navigation routes, etc.) is initially recommended. Assume the initial state is... The system recommends actions in this state. .

[0260] Feedback signal collection: User actions on recommended content (such as changing music, modifying navigation routes, etc.) are considered feedback signals. Feedback signals can be collected in several ways:

[0261] The frequency and duration of music switching.

[0262] User interactions with recommended content, such as adjusting volume or repeating playback.

[0263] Replanning or modifying navigation routes.

[0264] Reinforcement learning model training: Based on user feedback signals, the system employs the Deep Q-Learning (DQN) algorithm to optimize the recommendation strategy. The Q-value update formula is:

[0265] ;

[0266] in, γ is the learning rate, and γ is the discount factor. Reward users for their feedback on recommended content based on their current state. Indicates the system in state Select action Recommended strategy value.

[0267] Real-time feedback adjustments: When users provide negative feedback on recommended content (such as frequently switching to or canceling recommendations), the system will lower the recommendation priority of similar content; conversely, positive user interaction with a certain type of content (such as increasing playback counts) will increase the recommendation priority of that category. The system can continuously adjust its recommendation strategy based on users' real-time needs, ensuring more accurate recommendations in the future.

[0268] Implementation Results: Through an adaptive recommendation mechanism based on reinforcement learning, the system can analyze user feedback behavior in real time and continuously improve the accuracy of recommended content and user experience through continuous optimization and learning. Compared to traditional recommendation methods based on historical data, this mechanism can dynamically adapt to changes in users' immediate needs and preferences.

[0269] The existing technology in in-vehicle entertainment systems has the following main problems:

[0270] Single data source: Existing recommendation systems typically rely solely on users' operation history, behavior records, or voice commands, resulting in low recommendation accuracy and difficulty in meeting users' diverse needs.

[0271] Limited model complexity: The models in comparison documents 1, 2, and 3 are mostly based on simple algorithms and fail to fully consider the user's immediate needs and emotional state.

[0272] The limited variety of interaction methods: Existing interaction methods lack multi-dimensional information integration, fail to effectively improve user experience, and make user operation complex and inconvenient.

[0273] Technical solutions and their effects

[0274] Multimodal sensing data acquisition module

[0275] Note: The system collects multimodal data from users in real time using multiple sensors (voice, facial expressions, eye tracking, driving behavior, etc.), solving the problem of a single data source.

[0276] Results: Compared to comparison files 1, 2, and 3, the system can dynamically capture the user's emotional and behavioral state, improving the accuracy and personalization of recommendations.

[0277] Intelligent emotion recognition module

[0278] Note: By analyzing multiple information sources (facial expressions, voice emotions, driving behavior), we can identify users' emotional states in real time and ensure that recommended content matches users' immediate needs.

[0279] Results: Compared with the static recommendation method in document 1, this module can adjust the recommendation strategy in real time, improving personalization and context adaptability.

[0280] Personalized recommendation model

[0281] Note: The system trains deep learning models (LSTM, attention mechanisms, etc.) based on multimodal data, taking into account users' historical preferences, real-time emotions, and driving status to provide more accurate recommendations.

[0282] Results: Compared with the simple recommendation algorithms in comparison documents 2 and 3, this model dynamically adjusts based on real-time sentiment analysis, significantly improving the accuracy of recommendations and user satisfaction.

[0283] Multimodal interaction system

[0284] Note: Supports multiple interaction methods such as voice, gesture, touch and eye tracking to enhance the user interaction experience.

[0285] Results: Compared to the comparison document 3, which is limited to voice input, this system provides a flexible interaction method, reduces the complexity of user operation, and improves the convenience of interaction.

[0286] Adaptive recommendation feedback module

[0287] Note: The system analyzes user feedback in real time and continuously optimizes the recommendation model through reinforcement learning to ensure that the recommendation strategy is highly matched with user needs.

[0288] Results: Unlike the delayed feedback mechanisms found in the comparison files, this module enables real-time learning and optimization, enhancing the self-learning capabilities of the recommendation system.

[0289] In summary, the core technologies of this application are:

[0290] 1. Multimodal perception data acquisition and processing: Real-time acquisition and analysis of user status and driving environment through multimodal data such as voice, facial expressions, eye tracking, driving behavior, and vehicle status.

[0291] 2. Intelligent Emotion Recognition: Based on multimodal data, the system can identify the user's emotional state (such as tension, fatigue, etc.) in real time and infer the user's immediate needs in combination with the driving environment.

[0292] 3. Personalized Recommendation Model: Using deep learning models (such as LSTM and attention mechanisms) to integrate users' historical preferences, current emotions, and driving status to generate personalized and contextualized recommendation content.

[0293] 4. Adaptive Recommendation Feedback: Through reinforcement learning, the system can continuously optimize the recommendation strategy and adjust the recommended content in real time based on user feedback.

[0294] 5. Multimodal interaction system: Supports multiple interaction methods such as voice, gesture, touch screen and eye control, allowing users to freely choose the most suitable interaction method according to the driving environment.

[0295] Figure 9 This is an electronic device structural block diagram of a multi-modal personalized recommendation method for an in-vehicle entertainment system provided by one or more embodiments of the present invention.

[0296] like Figure 9 As shown, this application provides an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0297] The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the multimodal personalized recommendation method of the in-vehicle entertainment system.

[0298] This application also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of a multimodal personalized recommendation method for an in-vehicle entertainment system.

[0299] This application also provides a cockpit system, including:

[0300] Electronic devices, steps for implementing a multimodal personalized recommendation method for in-vehicle entertainment systems;

[0301] The processor runs a program that, when running, executes the steps of the multimodal personalized recommendation method of the in-vehicle entertainment system from data output by the electronic device.

[0302] Storage medium for storing programs that, when running, execute the steps of a multimodal personalized recommendation method of an in-vehicle entertainment system based on data output from electronic devices.

[0303] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0304] The electronic device comprises a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory. The operating system can be any one or more computer operating systems that control the electronic device through processes, such as Linux, Unix, Android, iOS, or Windows. Furthermore, in this embodiment of the invention, the electronic device can be a smartphone, tablet computer, or other handheld device, or a desktop computer, portable computer, or other electronic device; there is no particular limitation in this embodiment.

[0305] In this embodiment of the invention, the executing entity for electronic device control can be an electronic device itself, or a functional module within an electronic device capable of calling and executing a program. The electronic device can obtain the firmware corresponding to the storage medium. This firmware is provided by the supplier, and different storage media may have the same or different firmware; no limitation is made here. After obtaining the firmware corresponding to the storage medium, the electronic device can write this firmware into the storage medium; specifically, it burns the firmware corresponding to the storage medium into the storage medium. The process of burning the firmware into the storage medium can be implemented using existing technology, and will not be elaborated upon in this embodiment of the invention.

[0306] Electronic devices can also obtain reset commands corresponding to the storage media. The reset commands corresponding to the storage media are provided by the supplier. The reset commands corresponding to different storage media can be the same or different, and no restrictions are imposed here.

[0307] At this time, the storage medium of the electronic device is a storage medium on which the corresponding firmware has been written. The electronic device can respond to the reset command corresponding to the storage medium on which the corresponding firmware has been written, thereby resetting the storage medium on which the corresponding firmware has been written according to the reset command. The process of resetting the storage medium according to the reset command can be implemented by existing technology and will not be described in detail in this embodiment of the invention.

[0308] For ease of description, the above devices are described separately by function as various units and modules. Of course, in implementing this application, the functions of each unit and module can be implemented in one or more software and / or hardware.

[0309] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0310] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0311] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0312] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal personalized recommendation method for an in-vehicle entertainment system, characterized by, The multi-modal personalized recommendation method of the vehicle-mounted entertainment system comprises the following steps: multi-modal perception data acquisition and preprocessing; multi-modal data fusion and feature extraction; intelligent emotion recognition and demand inference; personalized recommendation content generation; multi-modal interaction response; adaptive recommendation optimization; The multi-modal perception data acquisition and preprocessing step comprises collecting multi-modal perception data and preprocessing the collected multi-modal perception data respectively, thereby providing rich data support for the recommendation system; The multi-modal data fusion and feature extraction step comprises using a long short-term memory network to process multi-modal time series data; The multi-modal data within a time window is set as an input vector, and the forgetting gate, input gate, cell state update, and output gate of the LSTM are used to calculate the time series change pattern of the user behavior and emotion, thereby generating a fusion feature vector; The intelligent emotion recognition and demand inference step comprises analyzing user state information based on the fusion feature vector and recognizing the user emotional state; According to the user emotional state, the user demand is inferred in combination with the vehicle driving situation; The personalized recommendation content generation step comprises constructing a personalized recommendation model, inputting user historical preference data, emotional state, and vehicle driving situation; The LSTM is used to capture the time series change of user preferences and adjust the weight of each input dimension through an attention mechanism, thereby generating vehicle-mounted entertainment recommendation content adapted to the current situation; The multi-modal interaction response step comprises completing the interactive control and feedback of the recommendation content based on the user-selected interaction mode; The adaptive recommendation optimization step comprises collecting the feedback behavior of the user to the recommendation content in real time; The feedback behavior process of the recommendation content is modeled as a Markov decision process; A deep Q-learning algorithm is used to update the recommendation strategy.

2. The method of claim 1, wherein, The multi-modal perception data acquisition and preprocessing step further comprises The multi-modal perception data acquisition comprises collecting user voice commands, facial expressions, eye movement trajectories, driving behavior data, and vehicle driving state data in real time through a voice recognition module, a facial recognition module, an eye movement tracking module, a driving behavior analysis module, and a vehicle state monitoring module; Further comprising obtaining in-vehicle and external environment data; The preprocessing of the collected multi-modal perception data respectively comprises extracting mel-frequency cepstral coefficients based on voice data; Based on facial expression data, facial key points are extracted through a convolutional neural network; Based on driving behavior data, noise is eliminated through Kalman filtering; Further comprising standardizing vehicle state data through a vehicle-mounted bus.

3. The method of claim 1, wherein, The multi-modal data fusion and feature extraction step further comprises using a long short-term memory network to process multi-modal time series data; The multi-modal data within a time window is set as an input vector, which comprises voice features, facial expression features, driving behavior features, and vehicle state features; Feature extraction comprises dynamically adjusting the weight of each modal data through an attention mechanism; The attention score of each modal feature is calculated, the weight is obtained after standardization by a Softmax function, and the fusion feature vector is generated through weighted summation.

4. The method of claim 1, wherein, The intelligent emotion recognition and demand inference step further comprises ​ The user's facial expression, voice emotion and driving behavior are analyzed to identify the user's emotional state; The demand inference includes inferring the user's immediate demand in combination with the vehicle driving situation.

5. The method of claim 1, wherein, The personalized recommendation content generation step further includes constructing a personalized recommendation model, inputting user historical preference data, emotional state and vehicle driving situation; The input user historical preference data includes music playback records and navigation habits; The in-vehicle entertainment recommendation content adapted to the current situation includes music, navigation route and video.

6. The method of claim 1, wherein, The multi-modal interaction response step further includes, The user-selected interaction mode includes voice, gesture, touch screen, eye movement control; The interaction control and feedback of the recommended content are completed by natural language processing to identify voice instructions, computer vision to identify gesture actions, optical flow method to track eye movement trajectories or / and event-driven UI to respond to touch operations.

7. The method of claim 1, wherein, The adaptive recommendation optimization step further includes, Real-time collection of user feedback behavior on the recommended content includes positive feedback and negative feedback; Positive feedback includes no modification, repeated playback and volume increase; Negative feedback includes switching content, modifying navigation and lowering volume; The feedback behavior process of the recommended content is modeled as a Markov decision process; The deep Q-learning algorithm is used to update the recommendation strategy, including setting reward values based on feedback behavior, optimizing recommendation model parameters through Q-value update formula, dynamically adjusting subsequent recommended content until the user's demand is met or the user terminates the interaction.

8. A multi-modal personalized recommendation system for an in-vehicle entertainment system, characterized in that, The multi-modal personalized recommendation system of the in-vehicle entertainment system includes: a multi-modal perception data acquisition and preprocessing module; a multi-modal data fusion and feature extraction module; an intelligent emotion recognition and demand inference module; a personalized recommendation content generation module; a multi-modal interaction response module; an adaptive recommendation optimization module; The multi-modal perception data acquisition and preprocessing module is used to acquire multi-modal perception data and preprocess the acquired multi-modal perception data respectively to provide rich data support for the recommendation system; The multi-modal data fusion and feature extraction module is used to process multi-modal time series data using a long short-term memory network; The multi-modal data in the time window is set as the input vector, and the forgetting gate, input gate, cell state update and output gate of the LSTM are calculated to capture the time series change pattern of user behavior and emotion, generating a fusion feature vector; The intelligent emotion recognition and demand inference module is used to analyze user state information based on the fusion feature vector to identify the user's emotional state; According to the user's emotional state, the user's demand is inferred in combination with the vehicle driving situation; The personalized recommendation content generation module is used to construct a personalized recommendation model, input user historical preference data, emotional state and vehicle driving situation; The LSTM captures the time series change of user preferences and adjusts the weight of each input dimension through an attention mechanism to generate in-vehicle entertainment recommendation content adapted to the current situation; The multi-modal interaction response module is used to complete the interaction control and feedback of the recommended content based on the user-selected interaction mode; The adaptive recommendation optimization module is used to collect user feedback behavior on the recommended content in real time; The feedback behavior process of the recommended content is modeled as a Markov decision process; A deep Q-learning algorithm is used to update the recommendation policy.

9. An electronic device, comprising: The method comprises the following steps: The processor, the communication interface, and the memory are in communication with each other through the communication bus; The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multi-modal personalized recommendation method of the in-vehicle entertainment system according to any one of claims 1 to 8.

10. A cockpit system characterized by, The method comprises the following steps: An electronic device is used to execute the steps of the multi-modal personalized recommendation method of the in-vehicle entertainment system according to any one of claims 1 to 8; A processor runs a program, and when the program runs, the data output from the electronic device executes the steps of the multi-modal personalized recommendation method of the in-vehicle entertainment system according to any one of claims 1 to 8; A storage medium is used to store a program, and when the program runs, the data output from the electronic device executes the steps of the multi-modal personalized recommendation method of the in-vehicle entertainment system according to any one of claims 1 to 8.

Images