Vehicle-mounted function control method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a vehicle-mounted function control method and device, equipment and a medium. The method comprises the steps of obtaining multi-modal user data collected by vehicle-mounted equipment, performing emotion recognition on the user data to obtain corresponding emotion features, inputting the emotion features into an emotion cognitive structure model for processing to obtain emotion state information of a user, and controlling a vehicle-mounted function according to the emotion state information. Complex emotion recognition is carried out based on the emotion cognitive structure model by using multi-modal user data, the emotion recognition accuracy is improved, the vehicle-mounted function is controlled according to the emotion state information, the human-vehicle interaction function is enriched, emotional services are provided, and the interaction experience and satisfaction degree of the user are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a vehicle-mounted function control method, a vehicle-mounted function control device, an electronic device and a computer-readable storage medium. Background Art

[0002] At present, the smart car industry is developing rapidly, and the popularity of smart cockpits with more and more functions in new models is increasing. With the continuous improvement of cockpit chip hardware and software and artificial intelligence related technologies, consumers are increasingly accepting the various functions equipped in the smart cockpit as the third space. A key technology in the smart cockpit is the interaction between the car and the driver and passengers. The interactive ability in the smart cockpit reflects its degree of intelligence to a certain extent.

[0003] Driver Monitoring System (DMS) / Occupant Monitoring System (OMS) have almost become standard equipment for mid- to high-end models. Currently, some DMS can monitor the driver's fatigue status and basic emotional state, and more comprehensive products can make some prompts based on the monitoring status.

[0004] However, current cockpit products cannot accurately judge and predict driver emotions, let alone make intelligent interactive decisions based on the environment and emotions. The system has false positives or missed positives. For example, it mistakenly considers the driver's checking the rearview mirror as distraction. Existing systems mainly focus on simple states (such as tiredness and distraction), and have weak recognition capabilities for complex emotions (such as stress, anger, anxiety, etc.), and can only recognize two categories: positive / negative. Most systems lack the ability to adapt to individual differences, and usually use fixed rules to process emotional states, failing to fully consider the personalized needs and emotional interactions of different users.

[0005] The industry has also begun to use the latest artificial intelligence algorithms such as big models to assist visual systems in interaction and decision-making output, which has brought new improvements to interaction and intelligence. However, current products generally lack situational awareness, emotion recognition is not accurate enough, and the response speed adjustment speed is slow, and they lack personalized and emotional interaction capabilities. Summary of the invention

[0006] In view of the above problems, embodiments of the present invention are proposed to provide a vehicle function control method, a corresponding vehicle function control device, an electronic device and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0007] In order to solve the above problems, an embodiment of the present invention discloses a vehicle function control method, the method comprising:

[0008] Obtain multi-modal user data collected by vehicle-mounted equipment;

[0009] Performing emotion recognition on the user data respectively to obtain corresponding emotion features;

[0010] Inputting the emotional characteristics into an emotional cognitive structure model for processing to obtain the emotional state information of the user;

[0011] The vehicle functions are controlled according to the emotional state information.

[0012] Optionally, the method further comprises:

[0013] Obtain external environment data collected by vehicle-mounted equipment;

[0014] The step of inputting the emotion feature into the emotion cognitive structure model for processing to obtain the emotion state information of the user includes:

[0015] The emotional characteristics and the external environment data are input into the emotional cognitive structure model for processing to obtain the emotional state information of the user.

[0016] Optionally, performing emotion recognition on the user data respectively to obtain corresponding emotion features includes:

[0017] respectively inputting the user data into an emotion recognition model;

[0018] The user data is identified based on the emotion recognition model to obtain corresponding emotion features.

[0019] Optionally, the identifying the user data based on the emotion recognition model to obtain corresponding emotion features includes:

[0020] Based on the emotion recognition model and historical data, the user data is identified to obtain corresponding emotion features.

[0021] Optionally, controlling the vehicle function according to the emotional state information includes:

[0022] The environment control system in the cockpit is controlled according to the emotional state information.

[0023] Optionally, controlling the vehicle function according to the emotional state information includes:

[0024] User interaction information is provided based on the emotional state information.

[0025] Optionally, controlling the vehicle function according to the emotional state information includes:

[0026] The vehicle functions are controlled according to the emotional state information and the personalized information of the user.

[0027] The embodiment of the present invention further discloses a vehicle-mounted function control device, the device comprising:

[0028] A user data acquisition module, used to acquire multi-modal user data collected by the vehicle-mounted equipment;

[0029] A feature recognition module, used to perform emotion recognition on the user data to obtain corresponding emotion features;

[0030] An information processing module, used for inputting the emotion characteristics into an emotion cognitive structure model for processing to obtain the emotion state information of the user;

[0031] The function control module is used to control the vehicle functions according to the emotional state information.

[0032] Optionally, the device further comprises:

[0033] An environmental data acquisition module is used to acquire external environmental data collected by vehicle-mounted equipment;

[0034] The information processing module is specifically used to: input the emotional characteristics and the external environment data into the emotional cognitive structure model for processing to obtain the emotional state information of the user.

[0035] Optionally, the feature recognition module includes:

[0036] An input submodule, used to input the user data into the emotion recognition model respectively;

[0037] The recognition submodule is used to recognize the user data based on the emotion recognition model to obtain corresponding emotion features.

[0038] Optionally, the identification submodule includes:

[0039] The recognition unit is used to recognize the user data based on the emotion recognition model and historical data to obtain corresponding emotion features.

[0040] Optionally, the function control module includes:

[0041] The system control submodule is used to control the environment control system in the cockpit according to the emotional state information.

[0042] Optionally, the function control module includes:

[0043] The information providing submodule is used to provide user interaction information according to the emotional state information.

[0044] Optionally, the function control module includes:

[0045] The function control submodule is used to control the vehicle functions according to the emotional state information and the personalized information of the user.

[0046] The embodiment of the present invention further discloses an electronic device, including:

[0047] one or more processors; and

[0048] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the apparatus to perform any of the methods described above.

[0049] The embodiment of the present invention further discloses a computer-readable storage medium, wherein a computer program stored in the storage medium enables a processor to execute any of the methods described above.

[0050] According to an embodiment of the present invention, multimodal user data collected by an on-board device is acquired, emotion recognition is performed on the user data respectively to obtain corresponding emotion features, the emotion features are input into an emotion cognition structure model for processing to obtain emotion state information of the user, and on-board functions are controlled according to the emotion state information, so that multimodal user data can be used to perform complex emotion recognition based on the emotion cognition structure model, the accuracy of emotion recognition can be improved, and on-board functions can be controlled according to the emotion state information, thereby enriching human-vehicle interaction functions, providing emotional services, and enhancing user interaction experience and satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flowchart of a vehicle function control method embodiment of the present invention;

[0052] Figure 2 It is a schematic diagram of the emotion perception decision-making system;

[0053] Figure 3 It is a schematic diagram of a multimodal emotion-aware decision-making system;

[0054] Figure 4 It is a schematic diagram of the principle of the OCC model;

[0055] Figure 5 is a flowchart of a vehicle function control method embodiment of the present invention;

[0056] Figure 6 It is a schematic diagram of the behavioral decision framework based on emotion simulation;

[0057] Figure 7 It is a schematic diagram of an emotion perception decision system based on OCC;

[0058] Figure 8 is a structural block diagram of an embodiment of a vehicle-mounted function control device of the present invention;

[0059] Fig. 9 The present invention is a structural block diagram of an electronic device for vehicle function control according to an exemplary embodiment. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] Reference Figure 1 , shows a flow chart of the steps of an embodiment of a vehicle function control method of the present invention, the method may specifically include the following steps:

[0062] Step 101: Acquire multi-modal user data collected by the vehicle-mounted equipment.

[0063] In the embodiment of the present invention, the vehicle-mounted device includes a camera, a microphone, a heart rate sensor and other devices for real-time collection of user data. Specifically, any applicable vehicle-mounted device may be included, and the embodiment of the present invention does not limit this.

[0064] In the embodiment of the present invention, user data refers to data related to the user, such as image data of the user's facial expression, voice and intonation data of the user, physiological signals of the user (such as heart rate), etc., or any other applicable user data, which is not limited in the embodiment of the present invention. The user may be a driver or a passenger.

[0065] In an embodiment of the present invention, the vehicle-mounted device can collect multi-modal user data, for example, using cameras and sensor technology to collect facial expressions, voice intonation, and physiological signals (such as heart rate).

[0066] In an optional embodiment of the present invention, the user data includes at least one of the following: image data, voice data, and physiological data.

[0067] Image data refers to images captured by a camera, for example, image data of a driver or a passenger's face.

[0068] The voice data refers to the voice collected by a microphone, for example, audio data of the voice of a driver or a passenger.

[0069] Physiological data refers to physiological data collected through sensors, such as the heart rate of the driver or passenger, thermal imaging cameras to monitor the temperature of the face, and adding additional conductance skin response sensors (GSR) to monitor changes in the user's skin conductance.

[0070] In the embodiment of the present invention, after acquiring the user data, the user data may be pre-processed, for example, filtering and denoising the collected data to ensure data quality.

[0071] For example, the data processing and storage module is responsible for the processing, storage and management of multimodal user data to ensure the efficient use and security of data. Specifically, it includes: Data preprocessing unit: preprocessing such as cleaning and denoising of real-time collected data. Data storage unit: an efficient database system is used to store and manage emotional data. Data security module: ensure security and privacy protection during data storage and storage. Data cleaning and filtering: use big data processing technology to clean and filter the collected raw data. Distributed storage: use a distributed database system to improve the reliability and scalability of data storage. Encryption technology: use encryption technology to protect sensitive data and ensure user privacy.

[0072] Step 102: Perform emotion recognition on the user data to obtain corresponding emotion features.

[0073] In the embodiment of the present invention, emotion recognition is performed for various user data to obtain emotion features corresponding to the user data. Emotion features are emotion classifications that can be determined based on user data. For example, based on the user's facial expression, it can be identified whether the user is happy, sad, or neither happy nor sad. For another example, based on the user's heart rate, it can be identified whether the user is excited or calm. For another example, based on the user's voice, it can be identified whether the user is positive or negative.

[0074] In the embodiment of the present invention, a deep learning algorithm or the like can be used to identify emotional features from user data such as facial expressions, voice, heart rate, etc. Any applicable implementation method can be used, and the embodiment of the present invention does not limit this.

[0075] For example, facial expression recognition: Convolutional neural networks (CNNs) can be used to analyze facial images and identify key emotional features.

[0076] Another example is speech emotion recognition: using recurrent neural networks (RNN) or long short-term memory networks (LSTM) to process speech data and identify emotional features.

[0077] Another example is physiological signal analysis: analyzing emotional characteristics through indicators such as heart rate variability.

[0078] Step 103: input the emotion feature into the emotion cognition structure model for processing to obtain the emotion state information of the user.

[0079] In the embodiment of the present invention, similar to the theory of human natural emotion, the emotional cognitive structure model can be established from different angles. Some emotional theory models emphasize motivation and its influence on emotional process and behavior, while others simulate emotional generation based on the probability of event occurrence. For example, the OCC model is based on the rules for generating emotions and is called the mother model of artificial emotions. The OCC model is based on the cognitive evaluation theory of emotions and sets the rules for generating different emotions. In the OCC model, the emotion generation process is as follows: first, three groups of evaluation variables related to the emotion generation rules are determined, namely objects, standards, and events. Each group of evaluation variables is divided into "desirable" or "undesirable", and then the evaluation results of one or more groups are matched with the emotion generation rules. Different combinations of evaluation results are matched to different emotional states, thereby determining what kind of emotional state is currently experienced. Unlike many other emotional models, in the OCC model, the basis for the generation of emotions is not a basic emotion set or a large irrelevant group based on dimensional theory, but an evaluation standard for events and objects, thereby inducing emotions. Many subsequent emotional models are expanded from the OCC model. The model also provides emotion generation rules that support expansion.

[0080] For example, Figure 2 The schematic diagram of the emotion perception decision system shown in the figure obtains the emotion characteristics (Affective signal) through the recognition module and the perception module. The emotion characteristics are used as input, and the decision making module predicts and infers, calculates and processes them to obtain the output emotion state information. Based on the emotion state information, the vehicle functions are controlled to simulate human-like behavior and simulate emotion output. Figure 3 The schematic diagram of a multimodal emotion perception decision system shown in the figure identifies the emotion features (Affective signal) based on text input (user input text and other user data), audio signals, image signals, physiological signals and fusion data, etc., through LLM (Large Language Model) modeling and recognition. The emotion features are used as the input of the analysis unit, and the analysis unit can calculate and process the emotion cognition structure model through LLM to obtain output emotion state information, thereby controlling the vehicle function to perform voice simulation and emotional decision (Affective Decision) based on the emotion state information. Figure 4 The schematic diagram of the principle of the OCC model shown in FIG. 1 shows that environmental information refers to the input of the OCC model, namely, the emotional characteristics. Classification is performed based on the emotional characteristics, and the classification of the emotional state is finally determined according to the branches designed by cognitive psychology.

[0081] In the embodiment of the present invention, the above-mentioned various emotional features are input into the emotional cognitive structure model. The emotional cognitive structure model calculates and classifies the emotional features to generate specific emotional state information. The emotional state information includes information such as the type of emotion and the strength of the emotional state, which is not limited in the embodiment of the present invention.

[0082] For example, the OCC model based on cognitive psychology is used for emotion classification. The extracted emotion features are mapped to specific emotion categories, such as joy, anger, sadness, etc. The decision model coefficients are adjusted through the emotion output, and the final decision is output. The OCC model is implemented through the rule engine and reasoning algorithm to classify and analyze the input emotion features. In addition, the strength of the emotion state can be evaluated based on the intensity of the emotion features. The OCC model is based on cognitive psychology theory and has a solid theoretical foundation to ensure the scientificity and credibility of emotion computing. It provides a detailed emotion classification framework, which can accurately classify and evaluate the emotion state by analyzing the multi-dimensional information of events, behaviors and objects. This detailed classification helps the system understand and respond to the user's emotions more accurately. In addition, the OCC model can not only classify emotions, but also evaluate the intensity of emotions, thereby providing a more delicate and comprehensive emotion analysis and improving the system's emotion recognition and response capabilities.

[0083] In the embodiment of the present invention, multimodal user data is used for complex emotion recognition. The system can not only detect simple emotion features, but also use the emotion cognitive structure model to identify complex emotion states such as stress, anger, anxiety, etc.

[0084] Step 104: Control the vehicle functions according to the emotional state information.

[0085] In the embodiment of the present invention, the vehicle-mounted functions include environmental control functions, user interaction functions, etc., or other arbitrarily used vehicle-mounted functions, which are not limited in the embodiment of the present invention. Through fast data processing and decision algorithms, the vehicle-mounted functions are controlled in real time, such as adjusting the environment of the cockpit in real time.

[0086] In the specific implementation, in order to realize the above process, the system integration and interface module is responsible for the integration and coordination of various functional modules and provides interfaces with external systems. Specifically, it includes: System integration platform: realize the seamless integration of various functional modules to ensure the integrity and consistency of the system. API interface: provide interfaces with other vehicle systems or external services, support data interaction and function expansion. Middleware technology: use middleware technology to realize the integration and data exchange of various modules. Standardized API: design standardized API to support interoperability with other systems.

[0087] For example, the system uses the OCC model to carefully classify and evaluate the emotional states of drivers and passengers, and optimizes the emotional decision-making process by combining the emotional decision-making model. Through the recognition and analysis of emotional features, the system can simulate the user's emotional state in real time and make corresponding decisions based on these emotional states, thereby improving the interactive experience and safety of drivers and passengers. Based on the emotional theory of psychology and cognitive behavior, the machine is emotionally empowered, and a new interaction and decision-making system is proposed based on decision theory and the latest artificial intelligence algorithm framework. It greatly satisfies the iteration of intelligent technology and helps to better improve the user experience. By integrating the emotional decision-making mechanism of the OCC model, the system can achieve high-precision emotion recognition and emotion-based decision-making, achieve natural emotional interaction and personalized emotional response, and make human-computer interaction more natural and humane. The system can control vehicle functions, enable in-vehicle interaction, provide personalized services, enhance the user's interactive experience and satisfaction, and also provide new directions and ideas for the development of smart cockpit technology.

[0088] In an optional embodiment of the present invention, a specific implementation method of controlling an in-vehicle function according to the emotional state information may include: controlling an environmental control system in the cabin according to the emotional state information.

[0089] The environment control system includes cabin environment control functions such as lighting, music, and temperature, and may specifically include any applicable environment control function, which is not limited in the embodiments of the present invention.

[0090] Controlling the environmental control system according to the emotional state information may include any applicable implementation method, which is not limited in this embodiment of the present invention.

[0091] For example, based on the emotional state information, the environmental settings in the cabin can be adjusted, such as turning the lights on and off, the color, brightness, etc., playing or stopping music, which music to play, raising or lowering the temperature, etc.

[0092] The smart cockpit is equipped with a variety of sensors (such as cameras, microphones, heart rate sensors, etc.) and actuators. These devices can provide rich data support and enhance the accuracy and real-time performance of emotional computing. The rich configuration of sensors and actuators makes it possible to expand the system functions. At the same time, these sensors and actuators enable the system to adjust the in-car environment (such as lighting, temperature, music) in real time, provide personalized comfort experience based on the user's emotional state information, and improve user satisfaction and driving safety.

[0093] In an optional embodiment of the present invention, a specific implementation method of controlling the vehicle-mounted function according to the emotional state information may include: providing user interaction information according to the emotional state information.

[0094] The user interaction information includes user interaction information in various forms such as images, voices, and texts, and may specifically include any applicable user interaction information, which is not limited in the embodiment of the present invention.

[0095] Providing user interaction information based on the emotional state information may include any applicable implementation method, which is not limited in this embodiment of the present invention.

[0096] For example, based on the emotional state information, emotional information in voice or text is provided to interact with the user.

[0097] For example, based on the emotional state information, the intelligent assistant in the cockpit is emotionally empowered, so that the intelligent assistant has anthropomorphic emotions and makes interactive decisions based on them. Through emotional and anthropomorphic interactions, a deeper trust relationship can be established with the user. The user will feel the system's understanding and empathy for his or her emotional state, which increases trust and reliance on the system.

[0098] In an optional embodiment of the present invention, a specific implementation method of controlling the in-vehicle function according to the emotional state information may include: controlling the in-vehicle function according to the emotional state information and the user's personalized information.

[0099] The user's personalized information includes the user's preference data and historical data, such as the user's favorite music, the user's disliked light colors, the user's accustomed temperature, etc., and may specifically include any applicable personalized information, which is not limited in the embodiments of the present invention.

[0100] When controlling the in-vehicle functions, the user's emotional state information and personalized information can be combined to perform personalized control of the in-vehicle functions, thereby providing a personalized emotional response.

[0101] Analyze based on the user's personalized information. For example, play the user's favorite music when the user feels tired, and dim the lights when the user feels stressed. Based on the anthropomorphic emotional state information, it can interact with the user emotionally, enhance the user experience, and provide personalized and emotional interaction.

[0102] According to an embodiment of the present invention, multimodal user data collected by an on-board device is acquired, emotion recognition is performed on the user data respectively to obtain corresponding emotion features, the emotion features are input into an emotion cognition structure model for processing to obtain emotion state information of the user, and on-board functions are controlled according to the emotion state information, so that multimodal user data can be used to perform complex emotion recognition based on the emotion cognition structure model, the accuracy of emotion recognition can be improved, and on-board functions can be controlled according to the emotion state information, thereby enriching human-vehicle interaction functions, providing emotional services, and enhancing user interaction experience and satisfaction.

[0103] Reference Figure 5 , shows a flow chart of the steps of an embodiment of a vehicle function control method of the present invention, the method may specifically include the following steps:

[0104] Step 201, obtaining multi-modal user data and external environment data collected by the vehicle-mounted equipment.

[0105] In the embodiment of the present invention, in addition to user data, external environment data also needs to be collected. The external environment data includes traffic status, weather conditions, vehicle driving conditions, etc., or any other applicable external environment data, which is not limited in the embodiment of the present invention.

[0106] In an optional embodiment of the present invention, the external environment data includes at least one of the following: traffic conditions and weather conditions.

[0107] Traffic status refers to data related to traffic conditions, such as traffic jam conditions, traffic light conditions, or any other applicable traffic conditions, which are not limited in the embodiments of the present invention.

[0108] Weather conditions refer to data related to weather conditions, such as weather type, outside temperature, inside temperature, etc., or any other applicable weather conditions, which are not limited in the embodiments of the present invention.

[0109] Step 202: input the user data into the emotion recognition model respectively.

[0110] In an embodiment of the present invention, the emotion recognition model is a machine learning model that can recognize user data and obtain emotion features corresponding to the user data. The emotion recognition model can be a model capable of recognizing multiple types of user data, or a model capable of recognizing only one type of user data.

[0111] In one implementation, different types of user data need to be recognized by corresponding emotion recognition models. For example, for facial expressions, there is an emotion recognition model that specifically recognizes the emotional state of facial expressions, and for speech, there is an emotion recognition model that specifically recognizes the emotional state of speech.

[0112] Step 203: Identify the user data based on the emotion recognition model to obtain corresponding emotion features.

[0113] In the embodiment of the present invention, after the user data is input into the emotion recognition model, the emotion characteristics can be recognized. For example, in addition to the deep learning algorithm, traditional machine learning algorithms such as support vector machine (SVM) or random forest (RF) or hybrid algorithms of statistical learning and deep learning can also be used for emotion recognition.

[0114] In an optional embodiment of the present invention, a specific implementation method of identifying the user data based on the emotion recognition model to obtain the corresponding emotion characteristics may include: identifying the user data based on the emotion recognition model and historical data to obtain the corresponding emotion characteristics.

[0115] When the emotion recognition model recognizes user data, it can retrieve historical data and use the historical data to complete the recognition of user data, thereby determining the emotion characteristics faster and more accurately.

[0116] Historical data can include short-term historical data and long-term historical data. Short-term historical data can be used to store conversation context and support multiple rounds of conversations, while long-term historical data can be used to store user features, business data, etc., and is usually quickly accessed through technologies such as vector databases.

[0117] Emotion recognition model utilizing short-term historical data is a short-term memory technique that is usually used to store and process information in the current task or session. It is characterized by limited capacity and short information retention time. The following are some techniques for implementing short-term memory:

[0118] Cache: In computer science, a cache is an efficient storage mechanism for storing temporary data. Emotion recognition models can use cache to quickly access recently used data.

[0119] Recurrent Neural Networks (RNNs): RNNs, especially Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), are neural network architectures that process sequential data and are able to retain information over a short period of time.

[0120] Context Window: In natural language processing tasks, sentiment recognition models can use context windows to process and remember information in the current session.

[0121] The use of long-term historical data in emotion recognition models is a long-term memory technology that is usually used to store information that needs to be retained for a long time, usually involving more complex data structures and storage mechanisms. The following are some technologies for implementing long-term memory:

[0122] Database: Use a relational or non-relational database to store and retrieve long-term information. This method is suitable for data that needs to be stored persistently.

[0123] Knowledge Graphs: Knowledge graphs are a structured data model used to represent entities and their relationships, suitable for storing and retrieving complex knowledge.

[0124] File System: Stores information in files for long-term preservation and access.

[0125] Experience Replay in Reinforcement Learning: In reinforcement learning, experience replay is used to store past experiences so that they can be used repeatedly during training.

[0126] Vector Database: used to store and retrieve high-dimensional vector data, suitable for processing embedding and similarity search.

[0127] For example, Figure 6 The diagram of the behavior decision framework based on emotion simulation is shown in the figure. A behavior decision framework based on an artificial intelligence agent (AI Agent) is constructed. The AI ​​agent can simulate human-like behavior according to the environment and external input, simulate emotional output, enable decision-making, and achieve harmonious human-computer interaction. The role of the AI ​​agent in the entire behavior decision framework is to give the AI ​​agent human emotions and achieve more realistic anthropomorphic behavior output. Assume that the AI ​​agent is in a collective centered on driving behavior in a cockpit, and triggers corresponding emotions and decisions based on the current state, preference characteristics, behavioral norms, goals, and the results of communication with the external environment. At the same time, a threshold mechanism is introduced, and new emotional states can also be triggered through the accumulation of time and emotional intensity. Among them, according to the cognitive layer, the perception layer and the main memory module, the emotional characteristics are output, and then the emotional state information is determined by the OCC emotion (or emotion) decision module. Emotions are simulated through the theoretical model of OCC, and then the decision database is used to generate and output behavioral decisions for interaction with users. As shown Figure 7 A schematic diagram of an OCC-based emotion perception decision system is shown, wherein the analysis unit is a behavior decision framework (Emotional Agent) that includes an OCC emotion (or emotion) recognition model.

[0128] Step 204: input the emotional characteristics and the external environment data into an emotional cognitive structure model for processing to obtain the emotional state information of the user.

[0129] In an embodiment of the present invention, the above-mentioned various emotional features and external environment data are input into the emotional cognitive structure model. The emotional cognitive structure model calculates and classifies the emotional features and external environment data to generate specific emotional state information. The emotional state information includes information such as the type of emotion and the strength of the emotional state, which is not limited in the embodiment of the present invention.

[0130] Step 205: Control the vehicle functions according to the emotional state information.

[0131] In the embodiment of the present invention, the specific implementation method of this step can refer to the description in the above embodiment and will not be repeated here.

[0132] According to an embodiment of the present invention, multimodal user data and external environment data collected by an on-board device are acquired, and according to the type of the user data, the user data are respectively input into corresponding emotion recognition models, and the user data is recognized based on the emotion recognition model to obtain corresponding emotion features, and the emotion features and the external environment data are input into an emotion cognition structure model for processing to obtain the user's emotion state information, and the on-board functions are controlled based on the emotion state information, so that complex emotion recognition is performed based on the emotion cognition structure model using multimodal user data and external environment data, the accuracy of emotion recognition is improved, and the on-board functions are controlled based on the emotion state information, thereby enriching the human-vehicle interaction functions, providing emotional services, and enhancing the user's interaction experience and satisfaction.

[0133] Integrating external environmental data into the emotion cognitive structure model enhances the situational awareness function of the emotion cognitive structure model. For example, it can distinguish the driver's distraction caused by checking the rearview mirror or external factors, reduce false alarms, and further improve the accuracy of emotion recognition.

[0134] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0135] Reference Figure 8 The structure block diagram of an embodiment of a vehicle-mounted function control device of the present invention is shown, which may specifically include the following modules:

[0136] The user data acquisition module 301 is used to acquire multi-modal user data collected by the vehicle-mounted device;

[0137] A feature recognition module 302 is used to perform emotion recognition on the user data to obtain corresponding emotion features;

[0138] An information processing module 303 is used to input the emotion characteristics into an emotion cognitive structure model for processing to obtain the emotion state information of the user;

[0139] The function control module 304 is used to control the vehicle functions according to the emotional state information.

[0140] Optionally, the user data includes at least one of the following: image data, voice data, and physiological data.

[0141] Optionally, the device further comprises:

[0142] An environmental data acquisition module is used to acquire external environmental data collected by vehicle-mounted equipment;

[0143] The information processing module is specifically used to: input the emotional characteristics and the external environment data into the emotional cognitive structure model for processing to obtain the emotional state information of the user.

[0144] Optionally, the external environment data includes at least one of the following: traffic conditions and weather conditions.

[0145] Optionally, the feature recognition module includes:

[0146] An input submodule, used to input the user data into the emotion recognition model respectively;

[0147] The recognition submodule is used to recognize the user data based on the emotion recognition model to obtain corresponding emotion features.

[0148] Optionally, the identification submodule includes:

[0149] The recognition unit is used to recognize the user data based on the emotion recognition model and historical data to obtain corresponding emotion features.

[0150] Optionally, the function control module includes:

[0151] The system control submodule is used to control the environment control system in the cockpit according to the emotional state information.

[0152] Optionally, the function control module includes:

[0153] The information providing submodule is used to provide user interaction information according to the emotional state information.

[0154] Optionally, the function control module includes:

[0155] The function control submodule is used to control the vehicle functions according to the emotional state information and the personalized information of the user.

[0156] According to an embodiment of the present invention, multimodal user data collected by an on-board device is acquired, emotion recognition is performed on the user data respectively to obtain corresponding emotion features, the emotion features are input into an emotion cognition structure model for processing to obtain emotion state information of the user, and on-board functions are controlled according to the emotion state information, so that multimodal user data can be used to perform complex emotion recognition based on the emotion cognition structure model, the accuracy of emotion recognition can be improved, and on-board functions can be controlled according to the emotion state information, thereby enriching human-vehicle interaction functions, providing emotional services, and enhancing user interaction experience and satisfaction.

[0157] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] Fig. 9 1 is a block diagram of an electronic device 600 for vehicle function control according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0159] Reference Fig. 9 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0160] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned vehicle function control. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0161] The memory 604 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0162] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0163] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0164] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0165] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0166] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of the components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0167] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0168] In an exemplary embodiment, the electronic device 600 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned shutdown control method.

[0169] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to complete the above-mentioned shutdown control method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0170] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables a server to execute a vehicle function control method, the method comprising:

[0171] Obtain multi-modal user data collected by vehicle-mounted equipment;

[0172] Performing emotion recognition on the user data respectively to obtain corresponding emotion features;

[0173] Inputting the emotional characteristics into an emotional cognitive structure model for processing to obtain the emotional state information of the user;

[0174] The vehicle functions are controlled according to the emotional state information.

[0175] Optionally, the user data includes at least one of the following: image data, voice data, and physiological data.

[0176] Optionally, the method further comprises:

[0177] Obtain external environment data collected by vehicle-mounted equipment;

[0178] The step of inputting the emotion feature into the emotion cognitive structure model for processing to obtain the emotion state information of the user includes:

[0179] The emotional characteristics and the external environment data are input into the emotional cognitive structure model for processing to obtain the emotional state information of the user.

[0180] Optionally, the external environment data includes at least one of the following: traffic conditions and weather conditions.

[0181] Optionally, performing emotion recognition on the user data respectively to obtain corresponding emotion features includes:

[0182] respectively inputting the user data into an emotion recognition model;

[0183] The user data is identified based on the emotion recognition model to obtain corresponding emotion features.

[0184] Optionally, the identifying the user data based on the emotion recognition model to obtain corresponding emotion features includes:

[0185] Based on the emotion recognition model and historical data, the user data is identified to obtain corresponding emotion features.

[0186] Optionally, controlling the vehicle function according to the emotional state information includes:

[0187] The environment control system in the cockpit is controlled according to the emotional state information.

[0188] Optionally, controlling the vehicle function according to the emotional state information includes:

[0189] User interaction information is provided based on the emotional state information.

[0190] Optionally, controlling the vehicle function according to the emotional state information includes:

[0191] The vehicle functions are controlled according to the emotional state information and the personalized information of the user.

[0192] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0193] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0194] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0195] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0196] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0197] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0198] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0199] The vehicle-mounted function control method and the vehicle-mounted function control device provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A vehicle function control method, characterized in that: The method comprises: Obtain multi-modal user data collected by vehicle-mounted equipment; Performing emotion recognition on the user data respectively to obtain corresponding emotion features; Inputting the emotional characteristics into an emotional cognitive structure model for processing to obtain the emotional state information of the user; The vehicle functions are controlled according to the emotional state information.

2. The method according to claim 1, characterized in that The method further comprises: Obtain external environment data collected by vehicle-mounted equipment; The step of inputting the emotion feature into the emotion cognitive structure model for processing to obtain the emotion state information of the user includes: The emotional characteristics and the external environment data are input into the emotional cognitive structure model for processing to obtain the emotional state information of the user.

3. The method according to claim 1, characterized in that The performing emotion recognition on the user data respectively to obtain corresponding emotion features includes: respectively inputting the user data into an emotion recognition model; The user data is identified based on the emotion recognition model to obtain corresponding emotion features.

4. The method according to claim 3, characterized in that The step of identifying the user data based on the emotion recognition model to obtain corresponding emotion features includes: Based on the emotion recognition model and historical data, the user data is identified to obtain corresponding emotion features.

5. The method according to claim 1, characterized in that The controlling of the vehicle-mounted functions according to the emotional state information includes: The environment control system in the cockpit is controlled according to the emotional state information.

6. The method according to claim 1, characterized in that The controlling of the vehicle-mounted functions according to the emotional state information includes: User interaction information is provided based on the emotional state information.

7. The method according to claim 1, characterized in that The controlling of the vehicle-mounted functions according to the emotional state information includes: The vehicle functions are controlled according to the emotional state information and the personalized information of the user.

8. A vehicle-mounted function control device, characterized in that: The method comprises: A user data acquisition module, used to acquire multi-modal user data collected by the vehicle-mounted equipment; A feature recognition module, used to perform emotion recognition on the user data to obtain corresponding emotion features; An information processing module, used for inputting the emotion characteristics into an emotion cognitive structure model for processing to obtain the emotion state information of the user; The function control module is used to control the vehicle functions according to the emotional state information.

9. An electronic device, characterized in that: include: one or more processors; and One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the apparatus to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer program stored therein enables the processor to execute the method according to any one of claims 1 to 7.

Images