Vehicle-mounted function active recommendation system and method, electronic equipment and computer readable medium

Abstract

The invention provides a vehicle-mounted function active recommendation system and method, and belongs to the field of automobiles. Performing space-time alignment on the multi-source data; the data fusion module is used for fusing the multi-source data to generate a scenarized joint emotion representation and acquiring an emotion category based on the joint emotion representation; constructing a multi-dimensional state space according to the emotion category, the vehicle state and the environmental risk index, and inputting the multi-dimensional state space into a strategy network to obtain a vehicle-mounted function recommendation strategy; and generating a service combination based on the vehicle-mounted function recommendation strategy. According to the technical scheme, active vehicle control function recommendation after multi-mode fusion and multi-mode emotion perception is achieved, the safety and intelligent experience of an intelligent vehicle are improved, the perception ability and cognitive ability of the model are greatly enhanced, the part of data is utilized, a recommendation mode is accurately decided, meanwhile, extremely high operability can be achieved, and the method is suitable for popularization and application. And the safety control of the vehicle can be greatly improved under some conditions.

Description

Technical Field

[0001] This invention relates to the field of automotive technology, and in particular to an in-vehicle function proactive recommendation system, method, electronic device, and computer-readable medium. Background Technology

[0002] In the field of emotion perception and proactive recommendation in automotive intelligent cockpits, some technologies have achieved initial commercial applications, but systemic solutions still need breakthroughs. Existing solutions, represented by US Patent US20230391310, mainly rely on single-modal vision or simple audio-visual fusion, triggering fixed vehicle control actions (such as dimming lights when fatigue is detected) through rule engines. These solutions have limitations such as poor environmental adaptability, rigid strategies, and high privacy risks. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art, and proposes an in-vehicle function proactive recommendation system, method, electronic device and computer-readable medium.

[0004] In a first aspect, embodiments of the present invention provide an in-vehicle function proactive recommendation system, comprising:

[0005] A multimodal signal acquisition module is used to acquire data from multiple sources;

[0006] The data and spatiotemporal synchronization module is used to align multi-source data in a spatiotemporal manner.

[0007] The cross-modal data fusion module is used to fuse multi-source data to generate a contextualized joint emotion representation and obtain emotion categories based on the joint emotion representation.

[0008] The reinforcement learning dynamic decision-making module constructs a multi-dimensional state space based on the emotion category, the vehicle state, and the environmental risk index, and inputs it into the policy network to obtain vehicle function recommendation strategies.

[0009] The service decision and dynamic recommendation module generates service combinations based on the in-vehicle function recommendation strategy.

[0010] In some embodiments, the multimodal signal acquisition module includes an in-vehicle camera, a microphone array, and a vehicle CAN bus;

[0011] The in-vehicle camera is used to capture facial expressions;

[0012] The microphone array is used to collect in-vehicle voice signals;

[0013] The vehicle CAN bus is used to read vehicle signals in real time.

[0014] In some embodiments, the data and spatiotemporal synchronization module uses a dynamic time warping algorithm to align the timestamps of multi-source data.

[0015] In some embodiments, the cross-modal data fusion module includes:

[0016] The feature extraction unit is used to extract facial micro-expression features and tone of voice features;

[0017] The cross-modal fusion unit is used to interact with visual, speech and vehicle state vectors to generate contextualized joint emotion representations;

[0018] The emotion classification output unit is used to output an emotion category based on the joint emotion representation.

[0019] The emotion categories include anger, sadness, fear, happiness, disgust, surprise, fatigue, and anxiety.

[0020] In some embodiments, the reinforcement learning dynamic decision-making module includes a hierarchical reward function, which is as follows:

[0021] R_t=a R_safety+b R_comfort+c R_efficiency, a+b+c=1, and 0.5≤a≤0.7, where:

[0022] R_safety is a safety reward that reflects driving safety.

[0023] R_comfort is a comfort reward, reflecting driving and riding comfort;

[0024] R_efficiency is an efficiency reward that reflects driving efficiency.

[0025] In some embodiments, the service decision and dynamic recommendation module develops an SOA interface based on the AUTOSAR AP platform, connects to the cockpit domain controller and chassis controller, and uses the dependency relationship between the graph attention network and the policy network to dynamically generate service combinations based on the current state.

[0026] In some embodiments, a privacy protection and security monitoring module is also included, which uploads encrypted feature vectors to the cloud via a federated learning framework.

[0027] Secondly, the present invention also provides a method for proactively recommending in-vehicle functions, for implementing the above system, comprising:

[0028] Collect data from multiple sources;

[0029] Spatiotemporal alignment of multi-source data;

[0030] Multi-source data is fused to generate contextualized joint emotion representations, and emotion categories are obtained based on these joint emotion representations.

[0031] A multi-dimensional state space is constructed using the emotion category, the vehicle status, and the environmental risk index, and then input into a policy network to obtain in-vehicle function recommendation strategies.

[0032] Service combinations are generated based on the aforementioned in-vehicle function recommendation strategy.

[0033] Thirdly, the present invention also provides an electronic device, comprising:

[0034] One or more processors;

[0035] Memory, used to store one or more programs;

[0036] When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0037] Thirdly, the present invention also provides a computer-readable medium on which a computer program is stored, which, when executed by a processor, implements the steps in the above-described method.

[0038] This invention provides an active vehicle function recommendation system that collects the driver's emotional characteristics and driving scenarios; spatiotemporally correlates the emotional characteristics and driving scenarios; fuses the correlated emotional characteristics and driving scenarios, and outputs an emotion category; constructs a multi-dimensional state space using the emotion category, vehicle state, and environmental risk index, and uses this multi-dimensional state space as input to a policy network, which outputs an in-vehicle function recommendation strategy; and generates service combinations based on the in-vehicle function recommendation strategy. This invention achieves active vehicle control function recommendation after multimodal fusion and multimodal emotion perception, which is beneficial for improving the safety and intelligent experience of intelligent vehicles, greatly enhancing the model's perception and cognitive abilities, and using this data to accurately determine recommendation methods. It also possesses strong operability and can significantly improve vehicle safety control in certain situations. Attached Figure Description

[0039] Figure 1 This is a structural block diagram of an embodiment of the in-vehicle function proactive recommendation system of the present invention;

[0040] Figure 2 This is a structural block diagram of an embodiment of the cross-modal data fusion module of the present invention;

[0041] Figure 3 This is a flowchart illustrating an embodiment of the in-vehicle function proactive recommendation method of the present invention;

[0042] Figure 4 This is a structural block diagram of an embodiment of the electronic device of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0044] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0045] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0046] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0047] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0048] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0049] In related technologies, the main approach relies on single-modal vision or simple audio-visual fusion, triggering fixed vehicle control actions (such as dimming lights when fatigue is detected) through a rule engine. This approach has limitations such as poor environmental adaptability, rigid strategies, and high privacy risks.

[0050] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides an in-vehicle function proactive recommendation system. Figure 1 This is a schematic diagram of the structure of an in-vehicle function active recommendation system provided in an embodiment of the present invention.

[0051] like Figure 1 As shown, the in-vehicle function proactive recommendation system includes:

[0052] The multimodal signal acquisition module 10 is used to acquire multi-source data.

[0053] In this embodiment, the multimodal signal acquisition module includes an in-vehicle camera, a microphone array, and a vehicle CAN bus. The in-vehicle camera is used to capture facial expressions, specifically the driver's facial expressions; for example, the in-vehicle camera can be an RGB+IR dual-mode camera. The microphone array is used for in-vehicle voice signals; for example, the microphone array uses 8-channel beamforming. The vehicle CAN bus is used to read CAN signals in real time, and the CAN signals include vehicle status data.

[0054] The vehicle status data includes vehicle speed, steering angle, braking status, and ADAS alarms.

[0055] Furthermore, for the in-vehicle camera, adaptive histogram equalization (CLAHE) is applied to the camera data to eliminate tunnel / backlight interference; the voice signal collected by the microphone array uses MVDR beamforming to suppress tire noise / wind noise; and the CAN bus data is smoothed by Savitzky-Golay filter.

[0056] The data and spatiotemporal synchronization module 20 is used to perform spatiotemporal alignment of multi-source data.

[0057] In this embodiment, the data and spatiotemporal synchronization module 20 uses the Dynamic Time Warping (DTW) algorithm to align the timestamps of multi-source data, ensuring that the time error between video, voice, and CAN signals is ≤50ms.

[0058] Specifically, an improved DTW algorithm (window constraint + multi-order derivative matching) is adopted. The improved DTW algorithm incorporates vehicle kinematic constraints (such as prioritizing the alignment of emotional features during rapid acceleration), reduces computational redundancy, and compresses the time synchronization error to ≤50ms, ensuring strict spatiotemporal correlation between emotion-related data (facial expressions captured by the in-vehicle camera and in-vehicle voice signals collected by the microphone array) and vehicle status (such as emergency braking / lane change).

[0059] The cross-modal data fusion module 30 is used to fuse multi-source data to generate a contextualized joint emotion representation and obtain emotion categories based on the joint emotion representation.

[0060] Please see Figure 2 The cross-modal data fusion module 30 includes: a feature extraction unit, a cross-modal fusion unit, and an emotion classification output unit.

[0061] The feature extraction unit includes extracting facial micro-expression features using a lightweight CNN (such as MobileNetV3); extracting acoustic features using a pre-trained speech model, specifically by first converting speech ASR to text, and then using a pre-trained speech model (such as Wav2Vec2) to extract acoustic features such as tone and speech rate; extracting vehicle signals from the CAN bus, and also obtaining vehicle location signals through Android Location (or corresponding modules of other systems when using other vehicle systems), and obtaining vehicle trajectory signals through the map pass-through SDK, thereby obtaining the vehicle status. The vehicle status has six dimensions, specifically including: 1) vehicle speed (real-time driving speed, affecting emergency braking and risk prediction), 2) acceleration / deceleration (reflecting the degree of aggressiveness of driving behavior), 3) steering angle (steering wheel operation angle, used to judge lane keeping ability), 4) braking status (braking frequency and force, associated with emergency situations), 5) driving duration (continuous driving time, positively correlated with fatigue), and 6) vehicle position / trajectory (GPS positioning and lane departure data, used for environmental risk analysis).

[0062] The cross-modal fusion unit is used to interact with visual, speech and vehicle state vectors to generate contextualized joint emotion representations.

[0063] This embodiment constructs a cross-modal Transformer and uses a multi-head attention mechanism to fuse visual, speech, and vehicle state data to generate a contextualized joint emotion representation. By using vehicle state as a contextual prior, the robustness of emotion recognition in high-speed and emergency scenarios is enhanced. Specifically, a learnable positional encoding is added to the Transformer, and the vehicle state vector is injected into the attention mechanism as a positional prior. In a preferred embodiment, the multimodal data also includes the driver's physiological signals, such as heart rate / skin conductivity, and fusing them with visual, speech, and vehicle state data can further improve the emotion recognition rate, reaching 90% accuracy (a 12% improvement over the existing 78%).

[0064] The emotion classification output unit outputs emotion categories based on the joint emotion representation. It simultaneously optimizes the regression of eight emotion categories (using cross-entropy loss) and emotion intensity, employing gradient normalization (GradNorm) to balance task weights and improve data recognition accuracy. The eight emotion categories are:

[0065] 1) Anger (a highly arousing, low-valence emotion that can easily lead to dangerous driving behavior).

[0066] 2) Sadness (a low-valence emotion that may lead to distraction).

[0067] 3) Fear (a highly arousing emotion that may affect driving decisions).

[0068] 4) Happiness (high-value emotion, usually positively correlated with safe driving).

[0069] 5) Dislike (related to driving environment or interactive experience).

[0070] 6) Surprise (sudden emotional outburst that may interfere with driving).

[0071] 7) Fatigue (low arousal state, significantly increasing the risk of accidents).

[0072] 8) Anxiety (moderate arousal level, prone to operational errors).

[0073] Specifically, the emotion classification output unit includes a classifier that takes the fused joint emotion representation as a feature vector input to the classifier. The classifier can output the probabilities of the above 8 emotion categories based on the probability distribution, and takes the emotion category with the highest probability as the output of the emotion classification output unit, and takes it as the driver's current emotional state.

[0074] The reinforcement learning dynamic decision-making module 40 constructs a multi-dimensional state space based on the emotion category, the vehicle state, and the environmental risk index, and uses this multi-dimensional state space as the input to the policy network, which can then output vehicle function recommendation strategies.

[0075] In this embodiment, the reinforcement learning dynamic decision-making module 40 specifically performs the following steps:

[0076] S1: Construct the state space.

[0077] It includes three dimensions:

[0078] 1. Emotional state (8 dimensions): Probability distribution vector from the output of the front-end fusion model [P(anger), P(sadness), P(fear), ..., P(anxiety)].

[0079] 2. Vehicle status (6 dimensions): vehicle speed, acceleration / deceleration, steering angle, braking status, driving duration, vehicle position / trajectory.

[0080] 3. Environmental Risk Index (Congestion / Collision Warning)

[0081] S2: Selection and Training of Reinforcement Learning Algorithms

[0082] The constructed multi-dimensional state space is input into the policy network, which uses a reinforcement learning dynamic decision engine to output in-vehicle function recommendation policies. The policy network is trained using a dual-delay deep deterministic policy gradient algorithm, enabling millisecond-level decision responses. For example, reinforcement learning can be used to optimize the encoding ability of large models for multimodal inputs (such as facial expressions, voice, and vehicle status). In tasks, reinforcement learning (RL) guides the large model to extract key facial expression features and combines them with voice commands to generate structured state representations. Reinforcement learning simulates environmental state transitions and reward predictions, making decision control methods that comply with privacy protection and security considerations.

[0083] Based on this, when the system detects "fatigue + vehicle speed ≥ 80km / h", it prioritizes safety intervention (seat vibration + AR-HUD warning) rather than simply playing music; in the "anxiety + congestion" scenario, it focuses on comfort adjustment (fragrance + white noise).

[0084] S3: For the in-vehicle function recommendation strategy, a hierarchical reward function is designed to evaluate it: The hierarchical reward function is as follows:

[0085] R_t=a R_safety+b R_comfort+c R_efficiency, a+b+c=1, and 0.5≤a≤0.7,

[0086] R_safety is a safety reward: it reflects driving safety and the contribution of decisions to driving safety.

[0087] R_comfort represents the comfort reward, reflecting the driving and riding comfort, and the contribution of the decision to driving and riding comfort;

[0088] R_efficiency is an efficiency reward that reflects driving efficiency and the optimization of driving efficiency by decision-making.

[0089] In the above-mentioned tiered reward function, the weight of safety reward is 0.5≤a≤0.7, which indicates that the safety weight accounts for a relatively high proportion. Moreover, the safety weight is dynamically adjusted according to the ADAS alarm level. That is, the higher the alarm level, the larger the value of a, and vice versa.

[0090] The service decision and dynamic recommendation module 50 is used to generate service combinations based on the in-vehicle function recommendation strategy.

[0091] In this embodiment, the service decision-making and dynamic recommendation module develops an SOA interface based on the AUTOSAR AP platform, connects to the cockpit domain controller and chassis controller, and uses the dependency relationship between the graph attention network and the policy network to dynamically generate service combinations based on the current state. Specifically, it constructs a vehicle control service knowledge base containing 5 major categories and 20+ functions (such as safety intervention, environmental adjustment, entertainment recommendations, etc.), and uses a graph neural network (GAT) to dynamically generate personalized service combinations. For example, in the scenario of "long-distance driving + pleasant mood", it recommends explanations of scenic spots along the way and seat ventilation; in the scenario of "nighttime highway + distraction detection", it triggers steering wheel vibration and instrument panel brightness adjustment. The interaction layer adopts non-intrusive interaction, specifically using AR-HUD dynamic prompts (such as service area icons overlaid on the navigation interface) and zoned haptic feedback to avoid the interference of traditional screen operation on driving attention and achieve "seamless" service recommendations.

[0092] It can be understood that the action a_t output by the intelligent agent is converted into specific control commands through the service-oriented architecture (SOA) of the AUTOSAR AP platform and then sent to the corresponding executor.

[0093] Cockpit area: air conditioning / seats / AR-HUD.

[0094] Chassis area: Steering wheel vibration / brake assist.

[0095] This forms a complete closed loop of "perception -> integration -> decision-making -> execution -> environmental change -> re-perception".

[0096] The system of this invention also includes a privacy protection and security monitoring module 60, which uploads encrypted feature vectors to the cloud through a federated learning framework. Specifically, the original data is not sent out; only the encrypted feature vectors (SHA-3 hash + homomorphic encryption) are uploaded. The cloud aggregates and updates the model, and through knowledge distillation (KL divergence constraint of teacher-student model), the multimodal sentiment model is compressed to within 100-200 million parameters, with an end-to-end inference latency of ≤300ms.

[0097] As can be understood, this embodiment uses a federated learning framework (FATE platform), where local devices only upload encrypted feature vectors (SHA-3 hash + homomorphic encryption), and the cloud aggregates and updates the global model. Through knowledge distillation (KL divergence constraint of teacher-student model), the multimodal sentiment model is compressed to within 100-200 million parameters, with an end-to-end inference latency of ≤300ms.

[0098] The in-vehicle function proactive recommendation system provided in the above embodiments of the present invention solves the problem of proactive vehicle control function recommendation after multimodal fusion and multimodal emotion perception, which is conducive to improving the safety and intelligent experience of intelligent vehicles, greatly enhancing the model's perception and cognitive abilities, and using this data to accurately decide on recommendation methods. At the same time, it has strong operability and can greatly improve the safe handling of vehicles in some cases.

[0099] Based on the same inventive concept, the present invention also provides a method for proactively recommending in-vehicle functions. Figure 3 This invention provides a flowchart of a method for proactively recommending in-vehicle functions, which is applied to the in-vehicle function proactive recommendation system provided in the above embodiments, and specifically includes:

[0100] Step S10: Collect multi-source data.

[0101] Furthermore, the multi-source data includes driver facial video data detected by the in-vehicle camera, in-vehicle voice data collected by the microphone array, and vehicle status data read from the CAN bus. The vehicle status data includes six dimensions, specifically: 1) vehicle speed (real-time driving speed, affecting emergency braking and risk prediction), 2) acceleration / deceleration (reflecting the aggressiveness of driving behavior), 3) steering angle (steering wheel operation angle, used to judge lane keeping ability), 4) braking status (braking frequency and force, associated with emergency situations), 5) driving duration (continuous driving time, positively correlated with fatigue), and 6) vehicle position / trajectory (GPS positioning and lane departure data, used for environmental risk analysis).

[0102] Step S20: Perform spatiotemporal alignment on the multi-source data.

[0103] In this embodiment, a dynamic time warping algorithm is used to align the timestamps of various multi-source data to ensure that the time error between video, audio, and vehicle status is ≤50ms.

[0104] It is understood that this embodiment addresses the asynchronous problem of multimodal data in traditional systems (frequency error between video images and vehicle signals acquired by the CAN bus) by using an improved DTW algorithm (window constraint + multi-order derivative matching) to compress the time synchronization error to ≤50ms, ensuring a strict spatiotemporal correlation between emotional features and driving scenarios (such as emergency braking / lane changing) and reducing computational redundancy.

[0105] Step S30: Fuse multi-source data to generate a contextualized joint emotion representation, and obtain the emotion category based on the joint emotion representation.

[0106] This embodiment constructs a cross-modal Transformer and uses a multi-head attention mechanism to fuse visual, speech, and vehicle state features to generate contextualized joint emotion representations.

[0107] Based on the joint emotion representation, the emotion category is output. The regression of 8 emotion categories (using cross-entropy loss) and emotion intensity is optimized simultaneously. Gradient normalization (GradNorm) is used to balance task weights and improve data recognition accuracy. The 8 emotion categories include:

[0108] 1) Anger (a highly arousing, low-valence emotion that can easily lead to dangerous driving behavior).

[0109] 2) Sadness (a low-valence emotion that may lead to distraction).

[0110] 3) Fear (a highly arousing emotion that may affect driving decisions).

[0111] 4) Happiness (high-value emotion, usually positively correlated with safe driving).

[0112] 5) Dislike (related to driving environment or interactive experience).

[0113] 6) Surprise (sudden emotional outburst that may interfere with driving).

[0114] 7) Fatigue (low arousal state, significantly increasing the risk of accidents).

[0115] 8) Anxiety (moderate arousal level, prone to operational errors).

[0116] This embodiment deploys a lightweight CNN to process facial micro-expressions, extracts intonation features from a pre-trained speech model, and generates a joint emotion representation through a cross-modal Transformer (multi-layer attention head). An innovative design is employed: vehicle state vectors (vehicle speed / steering angle / ADAS warning level) are introduced as contextual priors for modality fusion, improving emotion recognition accuracy by 12.7% in high-speed scenarios (≥80km / h) (compared to the baseline model).

[0117] Step S40: Construct a multi-dimensional state space using the emotion category, the vehicle state, and the environmental risk index, and input it into a policy network to obtain vehicle function recommendation strategies.

[0118] It is understood that this embodiment utilizes reinforcement learning to optimize the large model's encoding capabilities for multimodal inputs (such as visual, linguistic, and sensor data). In the task, RL guides the large model to extract key visual features from the environment and combines them with linguistic instructions to generate structured state representations. Reinforcement learning simulates environmental state transitions and reward predictions, making decision control methods that comply with privacy protection and security considerations.

[0119] Step S50: Generate a service combination based on the in-vehicle function recommendation strategy.

[0120] In this embodiment, an SOA interface is developed based on the AUTOSAR AP platform to connect to the cockpit domain controller and chassis controller. The dependency relationship between the graph attention network and the policy network is used to dynamically generate service combinations based on the current state.

[0121] This forms a complete closed loop of "perception -> integration -> decision-making -> execution -> environmental change -> re-perception".

[0122] The in-vehicle function proactive recommendation method provided in the above embodiments of the present invention solves the problem of proactive vehicle control function recommendation after multimodal fusion and multimodal emotion perception. This method is beneficial for improving the safety and intelligent experience of intelligent vehicles, greatly enhancing the model's perception and cognitive abilities, and using this data to accurately determine the recommendation method. It also possesses strong operability and, in some cases, can significantly improve vehicle safety and control.

[0123] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 4 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the in-vehicle function proactive recommendation methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0124] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0125] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0126] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0127] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the in-vehicle function proactive recommendation methods described in the above embodiments. The computer-readable storage medium can be volatile or non-volatile.

[0128] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described vehicle function proactive recommendation method.

[0129] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0130] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0131] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0132] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0133] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0134] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0135] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0136] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0138] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. An active recommendation system for in-vehicle functions, characterized by, It comprises: a multi-modal signal acquisition module for acquiring multi-source data; a data and space-time synchronization module for time-space alignment of multi-source data; a cross-modal data fusion module for fusing multi-source data to generate a scenario-based joint emotional representation, and obtaining an emotional category based on the joint emotional representation; a reinforcement learning dynamic decision module for constructing a multi-dimensional state space based on the emotional category, the vehicle state and the environmental risk index, and inputting a policy network to obtain a vehicle function recommendation strategy; a service decision and dynamic recommendation module for generating a service combination based on the vehicle function recommendation strategy.

2. The active recommendation system for vehicle functions according to claim 1, characterized in that: the multi-modal signal acquisition module comprises an in-vehicle camera, a microphone array and a vehicle CAN bus; the in-vehicle camera is used to capture facial expressions; the microphone array is used to acquire in-vehicle voice signals; the vehicle CAN bus is used to read vehicle signals in real time.

3. The system of claim 2, wherein, The data and space-time synchronization module aligns the timestamps of multi-source data using a dynamic time warping algorithm.

4. The system of claim 2, wherein, The cross-modal data fusion module comprises: a feature extraction unit for extracting facial micro-expression features and tone features; a cross-modal fusion unit for interacting visual, voice and vehicle state vectors to generate a scenario-based joint emotional representation; an emotional classification output unit for outputting an emotional category according to the joint emotional representation; The emotional category includes anger, sadness, fear, happiness, disgust, surprise, fatigue and anxiety.

5. The system of claim 1, wherein, The reinforcement learning dynamic decision module includes a hierarchical reward function, which is as follows: R_t = a R_safety + b R_comfort + c R_efficiency, a + b + c = 1, and 0.5 < a < 0.7, wherein: R_safety is a safety reward, reflecting driving safety; R_comfort is a comfort reward, reflecting driving comfort; R_efficiency is an efficiency reward, reflecting driving efficiency.

6. The system of claim 1, wherein, The service decision and dynamic recommendation module develops a SOA interface based on the AUTOSAR AP platform, accesses the cabin domain controller and the chassis controller, uses the dependency relationship between the graph attention network and the policy network, and dynamically generates a service combination according to the current state.

7. The system of claim 1, wherein, It also includes a privacy protection and security monitoring module that uploads encrypted feature vectors to the cloud through a federated learning framework.

8. A method of proactively recommending in-vehicle functions, characterized by, To implement the system of any one of claims 1-7, comprising: acquiring multi-source data; time-space alignment of multi-source data; fusing multi-source data to generate a scenario-based joint emotional representation, and obtaining an emotional category based on the joint emotional representation; constructing a multi-dimensional state space based on the emotional category, the vehicle state and the environmental risk index, and inputting a policy network to obtain a vehicle function recommendation strategy; generating a service combination based on the vehicle function recommendation strategy.

9. An electronic device, comprising: It comprises: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of claim 8.

10. A computer readable medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps in the method of claim 8. The computer program is executed by the processor to implement the steps in the method of claim 8.

Images