Intelligent cabin scene simulation and prediction method and system

Abstract

The invention belongs to the technical field of artificial intelligence, provides an intelligent cabin scene simulation and prediction method and system, and aims to solve the problem of low accuracy of intelligent cabin scene simulation and prediction in the prior art. When it is monitored that the driver exists in the preset intelligent cabin, collecting vehicle interior and exterior condition information data corresponding to the preset intelligent cabin, simulating vehicle interior and exterior scenes corresponding to the preset intelligent cabin according to the vehicle interior and exterior condition information data, and predicting a trend state corresponding to the current state of the driver based on the vehicle interior and exterior scenes; the target preset man-vehicle interaction mode adapted to the driver is predicted according to the trend state, and man-vehicle interaction with the driver is carried out based on the target preset man-vehicle interaction mode, so that the scene simulation and prediction accuracy of the intelligent cabin can be improved, and the driving safety of the driver is ensured on the premise of ensuring the driving safety of the driver. And the convenience and reliability of human-vehicle interaction are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and intelligent driving technology, and in particular to a method and system for simulating and predicting intelligent cockpit scenarios. Background Art

[0002] The smart cockpit refers to the in-vehicle space equipped with advanced software and hardware systems, which features human-machine interaction, Internet-connected services, multi-scenario applications, and extended functions. Smart cockpit scenario simulation and prediction is an advanced system based on artificial intelligence, big data, and the Internet of Things. It aims to provide personalized, proactive interactive experience for passengers in the vehicle through real-time perception, simulation analysis, and prediction of user needs. Smart cockpit scenario simulation and prediction can provide passengers in the vehicle with intimate services including but not limited to personalized services and safety enhancements, providing users with a good car experience.

[0003] In traditional technologies, scenario simulation and prediction of smart cockpits are generally performed through including but not limited to unimodal or multimodal perception, combined with deep learning models to simulate and predict smart cockpits, so as to provide users with considerate car services. Among them, unimodal means that the smart cockpit only relies on a single type of data for scenario simulation and prediction, for example, only using a camera to recognize facial expressions to judge emotions, or only using a microphone to collect voice commands to control navigation. Multimodal means that the smart cockpit integrates multiple different types of data inputs to simulate and predict scenarios. For example, the smart cockpit captures the user's gestures through a camera, and uses a microphone to detect the user's voice. At the same time, it also uses detection including but not limited to pressure sensors to comprehensively judge and predict user needs through "vision + voice + touch". Among them, unimodality and multimodality each have their own advantages and disadvantages. Unimodality has low computing resource requirements and a small delay in single data stream processing, but poor fault tolerance and one-sided information. Multimodality integrates different types of information such as voice, vision, and touch. It can capture rich information, optimize interaction strategies, and has strong fault tolerance, but it has high computing resource requirements and large delays.

[0004] However, the inventors realized that in traditional technologies, due to the constantly changing conditions inside and outside the smart cockpit, the smart cockpit scenario simulation and prediction, whether using single mode or multi-mode for simulation and prediction, cannot adapt well to the complex and changeable conditions of the smart cockpit. Therefore, the smart cockpit scenario simulation and prediction cannot accurately match the complex and changeable conditions of the smart cockpit, resulting in low accuracy of the smart cockpit simulation and prediction.

[0005] Therefore, how to improve the accuracy of intelligent cockpit scenario simulation and prediction has become an urgent problem to be solved in the field of intelligent driving based on artificial intelligence. Summary of the invention

[0006] The technical problem solved by the present invention is to solve the problem of low accuracy in scenario simulation and prediction in the intelligent cockpit of intelligent driving.

[0007] To solve the above technical problem, the present invention provides the following technical solutions: Monitor whether there is a driver in the preset intelligent cockpit; if the above monitoring is yes, collect the information data of the vehicle interior and exterior conditions corresponding to the preset intelligent cockpit; according to the information data of the vehicle interior and exterior conditions, simulate the vehicle interior and exterior scenarios corresponding to the preset intelligent cockpit; based on the vehicle interior and exterior scenarios, predict the trend state corresponding to the current state of the driver; according to the trend state, predict the target preset vehicle-person interaction method suitable for the driver, and based on the target preset vehicle-person interaction method, perform vehicle-person interaction with the driver.

[0008] As a preferred solution of the intelligent cockpit scenario simulation and prediction method of the present invention, predicting the trend state corresponding to the current state of the driver based on the vehicle interior and exterior scenarios includes: obtaining the personal information data of the driver; constructing a user portrait corresponding to the driver according to the personal information data and the vehicle interior and exterior scenarios; collecting the current state corresponding to the driver; according to the user portrait and the current state, and based on a preset driving state prediction model, predicting the upcoming consecutive state corresponding to the current state, and obtaining the trend state corresponding to the current state of the driver.

[0009] The present invention also provides an intelligent cockpit scenario simulation and prediction system, including: a first monitoring module for monitoring whether there is a driver in the preset intelligent cockpit; a first collection module for collecting the information data of the vehicle interior and exterior conditions corresponding to the preset intelligent cockpit if the above monitoring is yes; a first simulation module for simulating the vehicle interior and exterior scenarios corresponding to the preset intelligent cockpit according to the information data of the vehicle interior and exterior conditions; a first prediction module for predicting the trend state corresponding to the current state of the driver based on the vehicle interior and exterior scenarios; a first interaction module for predicting the target preset vehicle-person interaction method suitable for the driver according to the trend state, and performing vehicle-person interaction with the driver based on the target preset vehicle-person interaction method.

[0010] Advantages of the present invention: By monitoring whether there is a driver in the preset intelligent cockpit, when it is detected that there is a driver in the preset intelligent cockpit, collect the information data of the vehicle interior and exterior conditions corresponding to the preset intelligent cockpit, and based on the information data of the vehicle interior and exterior conditions, simulate the vehicle interior and exterior scenarios corresponding to the preset intelligent cockpit. Then, based on the vehicle interior and exterior scenarios, predict the tendency state corresponding to the current state of the driver, and according to the tendency state, predict the target preset human-vehicle interaction method suitable for the driver. And based on the target preset human-vehicle interaction method, conduct human-vehicle interaction with the driver, which can improve the accuracy of intelligent cockpit scenario simulation and prediction, so that the human-vehicle interaction of intelligent driving can better adapt to the complex and changeable vehicle conditions inside and outside the vehicle where the intelligent cockpit is located and the constantly changing personal conditions of the driver, realize the intelligent adaptation of the human-vehicle interaction method between the driver and the vehicle based on the perception of the vehicle interior and exterior environment, improve the convenience and reliability of human-vehicle interaction on the premise of ensuring the driving safety of the driver, enhance the intelligent experience of intelligent driving, and enhance the core competitiveness of human-vehicle interaction in the era of intelligent driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a schematic flowchart of the intelligent cockpit scenario simulation and prediction method provided by an embodiment of the present invention; Figure 2 It is a schematic overall flowchart of the intelligent cockpit scenario simulation and prediction method provided by an embodiment of the present invention; Figure 3 It is a schematic first sub-flowchart of the intelligent cockpit scenario simulation and prediction method provided by an embodiment of the present invention; Figure 4 It is a schematic second sub-flowchart of the intelligent cockpit scenario simulation and prediction method provided by an embodiment of the present invention; Figure 5 It is a schematic block diagram of the intelligent cockpit scenario simulation and prediction system provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0012] In order to make the above objects, features and advantages of the present invention more obvious and understandable, the following detailed description of the specific embodiments of the present invention will be given in conjunction with the accompanying drawings of the specification. Obviously, the described embodiments are some embodiments of the present invention, rather than all embodiments.

[0013] An embodiment of the present invention provides an intelligent cockpit scenario simulation and prediction method and system. The method and system can be applied to devices including but not limited to in-vehicle terminals, in-vehicle consoles, servers, cloud platforms, etc., and are adopted when performing intelligent cockpit scenario simulation and prediction in scenarios including but not limited to intelligent driving scenarios.

[0014] In the face of the technical problem of low accuracy in the scenario simulation and prediction of the intelligent cockpit involved in traditional intelligent driving technologies, the inventors proposed the intelligent cockpit scenario simulation and prediction method according to the embodiments of the present invention. The core idea of the embodiments of the present invention is as follows: collect the condition information inside and outside the vehicle, simulate the scenarios inside and outside the vehicle according to the condition information, predict the tendency state of the driver according to the simulated scenarios inside and outside the vehicle, determine the target preset interaction method adapted to the tendency state of the driver according to the predicted tendency state, and based on the target preset interaction method, perform vehicle-human interaction with the driver, which can improve the accuracy of the intelligent cockpit scenario simulation and prediction, so that the vehicle-human interaction of intelligent driving can better adapt to the complex and changeable vehicle conditions inside and outside the intelligent cockpit and the constantly changing personal conditions of the driver, realize the intelligent self-adaptation of the vehicle-human interaction method between the driver and the vehicle based on the perception of the environment inside and outside the vehicle, improve the convenience and reliability of the vehicle-human interaction on the premise of ensuring the driving safety of the driver, enhance the intelligent experience of intelligent driving, and enhance the core competitiveness of the vehicle-human interaction in the era of intelligent driving.

[0015] The present invention will be described in detail below through specific embodiments.

[0016] Embodiment 1, please refer to Figure 1 And Figure 2 , Figure 1 is the flowchart of the intelligent cockpit scenario simulation and prediction method provided by the embodiments of the present invention, Figure 2 is the overall flowchart of the intelligent cockpit scenario simulation and prediction method provided by the embodiments of the present invention. As Figure 1 shown, in this embodiment, the method includes but is not limited to the following steps S11-S16: S11. Monitor whether there is a driver in the preset intelligent cockpit.

[0017] Explanatorily, since the embodiments of the present invention mainly achieve the self-adaptation of the vehicle-human interaction method between the driver and the vehicle based on improving the accuracy of the intelligent cockpit scenario simulation and prediction, therefore, first monitor whether there is a driver in the preset intelligent cockpit, that is, monitor whether there is a driver in the vehicle corresponding to the preset intelligent cockpit, which can be monitored by means including but not limited to portrait recognition by shooting the driver's seat with a camera, pressure sensors on the driver's seat, etc., to determine whether there is a driver in the preset intelligent cockpit.

[0018] S12. In the case where it is monitored that there is no driver in the preset intelligent cockpit, do not collect the condition information data inside and outside the vehicle corresponding to the preset intelligent cockpit, and continuously monitor whether there is a driver in the preset intelligent cockpit; S13. In the case where it is monitored that there is a driver in the preset intelligent cockpit, collect the condition information data inside and outside the vehicle corresponding to the preset intelligent cockpit.

[0019] Explanatorily, when it is detected that there is no driver in the preset intelligent cockpit, that is, there is no driver in the vehicle corresponding to the preset intelligent cockpit, the vehicle interior and exterior condition information data corresponding to the preset intelligent cockpit is not collected, that is, the adaptation of the human-vehicle interaction method between the driver and the vehicle based on the intelligent cockpit scenario simulation and prediction is not started, and it is continuously monitored whether there is a driver in the preset intelligent cockpit.

[0020] When it is detected that there is a driver in the preset intelligent cockpit, that is, the adaptation of the human-vehicle interaction method between the driver and the vehicle based on the intelligent cockpit scenario simulation and prediction is started, and the vehicle interior and exterior condition information data corresponding to the preset intelligent cockpit is collected. Among them, the vehicle interior and exterior condition information data includes vehicle interior condition information data and vehicle exterior condition information data. The vehicle interior condition information data represents the condition information of the vehicle interior seating space corresponding to the preset intelligent cockpit. The vehicle interior condition information data includes, but is not limited to, the information of the passengers in the vehicle and their behaviors, and the vehicle interior light information. Among them, the information of the passengers and their behaviors includes, but is not limited to, whether the passengers are talking to each other, whether they are listening to the radio, voice, playing games, etc., and the corresponding audio environment information. The collection of the vehicle interior condition information data can be carried out by using, including but not limited to, in-vehicle cameras, microphones, vibration sensors, etc. The vehicle exterior condition information data represents the surrounding environment information outside the vehicle corresponding to the preset intelligent cockpit. The vehicle exterior condition information data includes, but is not limited to, road condition information, vehicle condition information, pedestrian information, vehicle exterior light information, weather information. Among them, the road condition information data includes, but is not limited to, traffic light signal information, sign information, prompt information, construction information. The collection of the vehicle exterior condition information data can be carried out by using, including but not limited to, out-vehicle cameras, light sensors, network weather website data sources, etc.

[0021] S14. Simulate the vehicle interior and exterior scenarios corresponding to the preset intelligent cockpit according to the vehicle interior and exterior condition information data.

[0022] Explanatorily, according to the vehicle interior and exterior condition information data, and based on simulation technologies corresponding to, including but not limited to, digital twin, mixed reality (MR), and human-computer interaction simulation, simulate the vehicle interior and exterior scenarios corresponding to the preset intelligent cockpit, that is, adopt corresponding simulation technologies including but not limited to the above to simulate the vehicle interior and exterior scenarios corresponding to the preset intelligent cockpit. The vehicle interior and exterior scenarios include simulating the interior scenario and the exterior scenario of the vehicle. The interior scenario represents the visualized scene inside the vehicle corresponding to the preset intelligent cockpit. The interior scenario includes, but is not limited to, the passengers in the vehicle and their behavior information, and the interior lighting information. The interior scenario includes, but is not limited to, the dim light scenario, the bright light scenario, the conversation scenario, the high-speed driving scenario, the medium-speed driving scenario, and the low-speed driving scenario. The exterior scenario represents the visualized scene outside the vehicle corresponding to the preset intelligent cockpit. The vehicle exterior condition information data includes, but is not limited to, road condition information, vehicle condition information, pedestrian information, exterior lighting information, and weather information. The exterior scenario includes, but is not limited to, the dim light scenario, the bright light scenario, the traffic congestion scenario, the scenario with heavy traffic flow, the scenario with many pedestrians, and the rain and snow weather scenario.

[0023] S15. Based on the vehicle interior and exterior scenarios, predict the tendency state corresponding to the current state of the driver.

[0024] Explanatorily, the current state represents the state where the driver is currently located, such as the highly concentrated state and the relatively relaxed state. Different driving situations will correspond to different driving states. For example, during the traffic congestion stage, the driver will drive carefully with high concentration. When there are fewer vehicles on the road and it is not at a traffic intersection, relatively speaking, the driving attention does not need to be so highly concentrated. The tendency state, also known as the foreground state, represents the direction, trend, or tendency of the driver's driving state to occur. The tendency state is the next state corresponding to the current state. The tendency state includes, but is not limited to, the tendency states corresponding to the driver's actions, operations, and emotions. The current state and the tendency state respectively describe the corresponding driving states of the driver at different times. The current state and the tendency state can be the same or different.

[0025] Since the conditions inside and outside the vehicle of the driver are constantly changing, correspondingly, the attention or actions of the driver are also constantly changing, and thus the driving state of the driver is also constantly changing. Therefore, the human-vehicle interaction method that adapts to the driver is also constantly changing. Thus, in the traditional technology, whether the intelligent cockpit scenario simulation and prediction use single-modal or multi-modal to perform intelligent cockpit simulation and prediction, adopting a single human-vehicle interaction method, neither can well adapt to the complex and changeable vehicle conditions and the constantly changing state of the driver in the intelligent cockpit, resulting in the intelligent cockpit scenario simulation and prediction being unable to accurately match the road conditions, vehicle conditions and the state of the people in the vehicle, leading to a low accuracy of the intelligent cockpit simulation and prediction. However, in the embodiments of the present invention, by simulating the scenarios inside and outside the vehicle, predicting the tendency state of the driver according to the simulated scenarios inside and outside the vehicle, determining the target preset interaction method adapted to the tendency state of the driver according to the predicted tendency state, and performing human-vehicle interaction with the driver based on the target preset interaction method, so that the human-vehicle interaction method of intelligent driving continuously adapts to the changing driving state of the driver, so that the human interaction method better adapts to the complex and changeable vehicle conditions inside and outside the intelligent cockpit and the constantly changing personal conditions of the driver, realizing the intelligent self-adaptation of the human-vehicle interaction method between the driver and the vehicle based on the perception of the environment inside and outside the vehicle, not only can the accuracy of the intelligent cockpit scenario simulation and prediction be improved, but also the efficiency and effect of the human-vehicle interaction can be improved.

[0026] S16. Predict a target preset human-vehicle interaction method adapted to the driver according to the tendency state, and perform human-vehicle interaction with the driver based on the target preset human-vehicle interaction method.

[0027] Explanatorily, according to the tendency state, predict a target preset human-vehicle interaction method adapted to the driver, and perform human-vehicle interaction with the driver based on the target preset human-vehicle interaction method, where the target preset human-vehicle interaction method represents a preset human-vehicle interaction method that is a target and is predicted to potentially suit the tendency state of the driver. Exemplarily, when the driver is highly concentrated on driving, it is suitable to perform human-vehicle interaction with the driver by voice. In the context where the driver is talking with the people in the vehicle, or in the scenario where the driver is listening to music or the radio, it is suitable to perform human-vehicle interaction with the driver by gesture. In the case of relatively dim light inside the vehicle, it is suitable to perform human-vehicle interaction by touch, etc. Thus, with the complex and changeable vehicle conditions inside and outside the intelligent cockpit and the constantly changing personal conditions of the driver, the corresponding transformation and intelligent self-adaptation of the human-vehicle interaction method between the driver and the vehicle based on the perception of the environment inside and outside the vehicle are realized. On the premise of ensuring the driving safety of the driver, the convenience and reliability of the human-vehicle interaction are improved, and the intelligent experience of intelligent driving is enhanced.

[0028] In an embodiment of the present invention, by monitoring whether there is a driver in a preset intelligent cockpit, when it is detected that there is a driver in the preset intelligent cockpit, the information data of the vehicle interior and exterior conditions corresponding to the preset intelligent cockpit is collected, and based on the information data of the vehicle interior and exterior conditions, the vehicle interior and exterior scenarios corresponding to the preset intelligent cockpit are simulated. Then, based on the vehicle interior and exterior scenarios, the tendency state corresponding to the current state of the driver is predicted, and according to the tendency state, the target preset vehicle-human interaction method suitable for the driver is predicted. And based on the target preset vehicle-human interaction method, vehicle-human interaction is carried out with the driver, so that it is possible to predict the tendency states corresponding to the actions, operations, emotions, etc. that the driver will perform according to the different vehicle interior and exterior conditions where the intelligent cockpit is located, and according to the future tendency conditions, predict the potentially suitable convenient interaction methods for the driver, and then adopt the convenient interaction method adapted to the tendency state to carry out vehicle-human interaction between the driver and the vehicle, so as to realize vehicle-human interaction between the driver and the vehicle in a convenient interaction method for the driver without affecting the states of the driver including but not limited to actions, operations, emotions that will occur, which can improve the matching accuracy of intelligent cockpit scenario simulation and prediction and the complex and changeable conditions inside and outside the vehicle where the intelligent cockpit is located and the constantly changing conditions of the driver, thereby improving the accuracy of intelligent cockpit scenario simulation and prediction, realizing the intelligent adaptation of the vehicle-human interaction method between the driver and the vehicle based on vehicle interior and exterior environment perception, automatically providing the driver with a personalized, customized, changing, and actively convenient intelligent adaptive vehicle-human interaction method suitable for the tendency state, improving the vehicle-human interaction efficiency and effect of intelligent driving in complex and changeable conditions, improving the convenience and reliability of vehicle-human interaction on the premise of ensuring the driving safety of the driver, enhancing the intelligent experience of intelligent driving, and enhancing the core competitiveness of vehicle-human interaction in the era of intelligent driving.

[0029] In one embodiment, please refer to Figure 3 , Figure 3 which is the first sub-process schematic diagram of the intelligent cockpit scenario simulation and prediction method provided by the embodiment of the present invention. As Figure 3 shown, in this embodiment, based on the vehicle interior and exterior scenarios, predicting the tendency state corresponding to the current state of the driver includes: S31. Obtain the personal information data of the driver; S32. Construct a user portrait corresponding to the driver according to the personal information data and the vehicle interior and exterior scenarios; S33. Collect the current state corresponding to the driver; S34. According to the user portrait and the current state, and based on a preset driving state prediction model, predict the upcoming subsequent state corresponding to the current state to obtain the tendency state corresponding to the current state of the driver.

[0030] Explanatorily, a driving state prediction model is preset, that is, a preset driving state prediction model, which represents a model for predicting the next driving state of a driver. The preset driving state prediction model includes, but is not limited to, a recurrent neural network (RNN), a long short-term memory network (LSTM), a generative adversarial network (GAN), a Transformer model, and a GenAD model.

[0031] According to the above concept and setting, personal information data of the driver is obtained. The personal information data includes, but is not limited to, age, gender, driving experience, vision index, personality, emotion, and historical behavior information data. Based on the personal information data and the in-vehicle and out-of-vehicle scenarios, a user portrait corresponding to the driver is constructed. The user portrait represents a virtual character model of the driver constructed by collecting multi-dimensional data of the personal information data corresponding to the driver and the in-vehicle and out-of-vehicle scenarios. The virtual character model of the driver is used to accurately describe the characteristics, needs, reactions, and behavior patterns of the driver in the above in-vehicle and out-of-vehicle scenarios. Thus, the user portrait can describe the different thoughts, states, and different reactions of different drivers in different in-vehicle and out-of-vehicle scenarios, and collect the current state of the driver corresponding to the driver. The current state is the current state of the driver. Then, based on the user portrait and the current state, and based on the preset driving state prediction model, the upcoming consecutive state corresponding to the current state is predicted, and the tendency state corresponding to the current state of the driver is obtained. The tendency state is as described above.

[0032] Exemplarily, in the face of congested road conditions, for an experienced male driver, it may be very steady, for an inexperienced male driver, it may be relatively stable, and for an inexperienced female driver, it may be very nervous. Thus, the behavior performances of different drivers in different in-vehicle and out-of-vehicle scenarios are all different. After predicting the tendency state, according to the tendency state, the target preset vehicle-human interaction method suitable for the driver is predicted, which can make the vehicle-human interaction of intelligent driving better adapt to the complex and changeable vehicle conditions inside and outside the vehicle and the constantly changing personal conditions of the driver based on the accuracy of intelligent cockpit scenario simulation and prediction, and realize the intelligent self-adaptation of the vehicle-human interaction method between the driver and the vehicle based on the perception of the in-vehicle and out-of-vehicle environment. On the premise of ensuring the driving safety of the driver, the convenience and reliability of the vehicle-human interaction are improved.

[0033] Furthermore, obtaining the personal information data of the driver includes at least one of the following: Obtaining the personal profile information data of the driver; Collecting the multi-modal personality factor information data of the driver, and determining the personal personality information data of the driver based on a preset multi-modal fusion personality model; Collect the multi-modal emotion factor information data of the driver, and determine the personal emotion information data of the driver based on a preset multi-modal fusion emotion model; Obtain the historical behavior information data of the driver.

[0034] Specifically, a multi-modal fusion personality model is preset, that is, a preset multi-modal fusion personality model, which represents a model that fuses personality factor information of different modalities to predict the personality of a driver. The preset multi-modal fusion personality model includes, but is not limited to, a recurrent neural network (RNN), a long short-term memory network (LSTM), a generative adversarial network (GAN), a Transformer model, and a GenAD model. Similarly, a multi-modal fusion emotion model is preset, that is, a preset multi-modal fusion emotion model, which represents a model that fuses emotion factor information of different modalities to predict the emotion of a driver. The preset multi-modal fusion emotion model includes, but is not limited to, a recurrent neural network (RNN), a long short-term memory network (LSTM), a generative adversarial network (GAN), a Transformer model, and a GenAD model.

[0035] According to the above concept and setting, obtain the personal information data of the driver, including at least one of the following: 1) Obtain the personal profile information data of the driver, and the personal profile information data includes, but is not limited to, age, gender, driving experience, and vision index.

[0036] 2) Collect the multi-modal personality factor information data of the driver, and determine the personal personality information data of the driver based on the preset multi-modal fusion personality model. Among them, the multi-modal personality factor information data includes, but is not limited to, behavior modal personality factor information, physiological signal modal personality factor information, and speech and language modal personality factor information. The behavior modal personality factor information represents the personality factors and personality manifestations analyzed based on behavior information data. For example, the diversity of music or route selection represents an open personality, and the on-time arrival rate or vehicle maintenance frequency represents a conscientious personality, etc.; the physiological signal modal personality factor information represents the personality factors and personality manifestations analyzed based on physiological signals. For example, the galvanic skin response (GSR) indicates that the neuroticism level is positively correlated with the stress response intensity, which can be collected through a steering wheel capacitance film sensor. In the eye movement pattern, those with high openness have a 35% wider visual exploration range, which can be collected through an infrared camera (pupil diameter / fixation point); the physiological signal modal personality factor information represents the personality factors and personality manifestations analyzed based on speech features and language features. For example, extroverts have a fast speech rate (>4.3 syllables / second) and a large fundamental frequency change (ΔF0>40Hz), and high conscientiousness: the sentence structure is complete (the usage rate of conjunctions is 22% higher).

[0037] 3) Collect multimodal emotion factor information data of the driver, and determine the driver's personal emotion information data based on a preset multimodal fusion emotion model. The multimodal emotion factor information data includes but is not limited to physiological signal modality emotion factor information, visual modality emotion factor information, voice emotion modality emotion factor information, and behavioral modality emotion factor information.

[0038] Among them, the physiological signal modal emotion factor information represents the emotion factors and emotion manifestations based on physiological signal analysis. For example, galvanic skin response (GSR) detects emotional arousal through sweat gland activity, and heart rate variability (HRV) detects emotions through the state of the autonomic nervous system.

[0039] Visual modality emotion factor information represents emotion factors and emotion manifestations based on visual signal analysis. For example, the driver's emotions are detected through facial expression analysis, which can be collected and analyzed through 3D cameras + CNN (such as ResNet). The driver's emotions are detected through pupil dynamic tracking, which can be collected and analyzed through infrared cameras + pupil diameter change analysis.

[0040] The emotional factor information of speech emotional modality represents the emotional factors and emotional manifestations based on speech signal analysis. For example, the fundamental frequency (F0) increases by 30% in anger and decreases by 20% in sadness. The driver's emotions can be detected through cepstrum analysis (MFCC). The speaking speed will increase by 15%-25% when nervous, and the driver's emotions can be detected through syllable segmentation (DTW algorithm).

[0041] The behavioral modal emotional factor information represents the emotional factors and emotional manifestations based on behavioral pattern analysis. For example, the steering wheel grip increases by 3-5N when the driver is nervous, and the driver's emotions can be detected through the pressure sensor matrix. The probability of sudden acceleration increases by 70% when the driver is angry, and the driver's emotions can be detected through the throttle opening change rate. The body posture will lean back at a angle greater than 15° when the driver is frustrated, and the driver's emotions can be detected through ToF camera + skeleton tracking.

[0042] 4) Obtain the driver's historical behavior information data. The historical behavior information data represents the driver's historical status corresponding to different conditions inside and outside the car in the past and different conditions of the driver, such as the historical status when talking with people in the past, the historical status when the light is dim, etc. The historical behavior information data can better reflect the driver's reaction habits and action habits in different complex and changeable conditions inside and outside the car and the driver's changing conditions, which can further improve the accuracy of intelligent cockpit scenario simulation and prediction and the matching of the complex and changeable conditions inside and outside the car and the driver's changing conditions, thereby improving the accuracy of intelligent cockpit scenario simulation and prediction.

[0043] According to the above description, collecting the personal information data of the driver from different perspectives can improve the accuracy of constructing the user profile corresponding to the driver, thereby improving the accuracy of predicting the tendency state corresponding to the current state of the driver, and further improving the accuracy of intelligent cockpit scenario simulation and prediction.

[0044] In an embodiment of the present invention, by obtaining the personal information data of the driver and constructing the user profile corresponding to the driver according to the personal information data and the in-vehicle and out-of-vehicle scenarios, then collecting the current state corresponding to the driver, and then according to the user profile and the current state, and based on a preset driving state prediction model, predicting the upcoming consecutive state corresponding to the current state to obtain the tendency state corresponding to the current state of the driver. Thus, according to the external environment information corresponding to the in-vehicle and out-of-vehicle condition information data and the personal internal reaction information corresponding to the personal information of the driver, the above information from different dimensions is combined to predict the state manifested by the behavior that the driver will perform next. Since a person's behavior often stems from the stimulation of the external environment and the reaction reflected by personal internal cognition, therefore, by combining the above information from different dimensions to predict the tendency state corresponding to the current state of the driver, the prediction accuracy of the behavior that the driver is about to perform, that is, the accuracy of the tendency state prediction, can be improved. Furthermore, according to the tendency state, predicting the potentially suitable convenient interaction method for the driver, and then adopting the convenient interaction method adapted to the tendency state for the vehicle-person interaction between the driver and the vehicle can improve the accuracy of intelligent cockpit scenario simulation and prediction, so that the vehicle-person interaction of intelligent driving can better adapt to the complex and changeable vehicle conditions inside and outside the intelligent cockpit and the constantly changing personal conditions of the driver, realizing the intelligent adaptation of the vehicle-person interaction method between the driver and the vehicle based on the perception of the in-vehicle and out-of-vehicle environment, and improving the convenience and reliability of the vehicle-person interaction on the premise of ensuring the driving safety of the driver.

[0045] In one embodiment, collecting the in-vehicle and out-of-vehicle condition information data corresponding to the preset intelligent cockpit includes at least one of the following: Collecting the out-of-vehicle condition information data corresponding to the preset intelligent cockpit based on a preset first category information collection method; Collecting the in-vehicle condition information data corresponding to the preset intelligent cockpit based on a preset second category information collection method.

[0046] Explanatorily, a first category of information collection methods is preset, that is, a first category of information collection methods is preset. The preset first category of information collection methods represents the information collection methods for collecting the information data of the external conditions corresponding to the preset intelligent cockpit, and based on the preset first category of information collection methods, the information data of the external conditions corresponding to the preset intelligent cockpit is collected. Among them, the preset first category of information collection methods includes but is not limited to a preset first light collection device, a preset road condition collection device, and a preset weather collection method.

[0047] A second category of information collection methods is preset, that is, a second category of information collection methods is preset. The preset second category of information collection methods represents the information collection methods for collecting the information data of the internal conditions corresponding to the preset intelligent cockpit, and based on the preset second category of information collection methods, the information data of the internal conditions corresponding to the preset intelligent cockpit is collected. Among them, the preset second category of information collection methods includes but is not limited to a preset second light collection device, a preset audio collection device, and a preset image collection device.

[0048] Further, based on the preset first category of information collection methods, collecting the information data of the external conditions corresponding to the preset intelligent cockpit includes at least one of the following: Based on the preset first light collection device, collecting the information data of the external light corresponding to the preset intelligent cockpit; Based on the preset road condition collection device, collecting the information data of the external road conditions corresponding to the preset intelligent cockpit; Based on the preset weather collection method, collecting the information data of the external weather corresponding to the preset intelligent cockpit.

[0049] Specifically, light information, vehicle condition, road condition, traffic lights, signs, pedestrians, weather A preset first light collection device is preset, that is, a preset first light collection device. The preset first light collection device represents a device for collecting the external light information of the vehicle. The preset first light collection device includes but is not limited to an ambient light sensor (ALS), an RGB sensor, and a spectral sensor. Based on the preset first light collection device, the information data of the external light corresponding to the preset intelligent cockpit is collected. The information data of the external light represents the light information outside the vehicle, including but not limited to light intensity, light color temperature, and light direction. Among them, the "first" involved in the preset first light collection device is only used to distinguish different light collection devices and is not used to limit different light collection devices. The same applies to other similar descriptions in the embodiments of the present invention.

[0050] Pre-set road condition collection devices, that is, pre-set road condition collection devices. The pre-set road condition collection devices are used to collect information on the driving environment of the vehicle corresponding to the pre-set intelligent cockpit. The pre-set road condition collection devices include, but are not limited to, lidar (LiDAR), forward-looking cameras, and ultrasonic radars. Based on the pre-set road condition collection devices, collect the data of the external road condition information corresponding to the pre-set intelligent cockpit. The external road condition information data represents the information on the road conditions outside the vehicle corresponding to the intelligent cockpit. The external road condition information data includes, but is not limited to, road information, traffic flow information, and pedestrian flow information. Among them, the road information includes, but is not limited to, obstacle information, lane lines, traffic signs, traffic light signals, and construction information.

[0051] Pre-set weather collection methods, that is, pre-set weather collection methods. The pre-set weather collection methods represent the methods for collecting weather information. The pre-set weather collection methods include, but are not limited to, light sensors, rain sensors, temperature and humidity sensors, barometric pressure sensors, lidar assistance, camera + AI recognition, and direct connection to meteorological stations, and based on the pre-set weather collection methods, collect the data of the external weather information corresponding to the pre-set intelligent cockpit.

[0052] According to the above concepts, settings and collections, collect the information from different angles outside the vehicle corresponding to the pre-set intelligent cockpit to obtain the corresponding external condition information data.

[0053] Further, based on the pre-set second category of information collection methods, collect the data of the internal condition information corresponding to the pre-set intelligent cockpit, including at least one of the following: Based on the pre-set second light collection device, collect the data of the internal light information corresponding to the pre-set intelligent cockpit; Based on the pre-set audio collection device, collect the data of the internal audio information corresponding to the pre-set intelligent cockpit; Based on the pre-set image collection device, collect the data of the behavior information of the occupants inside the vehicle corresponding to the pre-set intelligent cockpit.

[0054] Specifically, pre-set the second light collection device, that is, the pre-set second light collection device. The pre-set second light collection device represents the device for collecting the light information inside the vehicle. The other contents of the pre-set second light collection device are the same or similar to those of the pre-set first light collection device, which will not be elaborated here. Based on the pre-set second light collection device, collect the data of the internal light information corresponding to the pre-set intelligent cockpit.

[0055] Pre-set an audio acquisition device, that is, a preset audio acquisition device, and based on the preset audio acquisition device, collect in-vehicle audio information data corresponding to the preset intelligent cockpit. The in-vehicle audio information data includes but is not limited to conversation sounds, broadcast sounds, and music sounds. Among them, the preset audio acquisition device refers to a device for collecting audio inside the vehicle, and the preset audio acquisition device includes but is not limited to a microphone array and a vibration sensor.

[0056] Pre-set an image acquisition device, that is, a preset image acquisition device, and based on the preset image acquisition device, collect in-vehicle occupant behavior information data corresponding to the preset intelligent cockpit. The in-vehicle occupant behavior information data includes but is not limited to conversations, actions, playing games, and watching movies and TV shows. Among them, the preset image acquisition device refers to a device for collecting images inside the vehicle, and the preset image acquisition device includes but is not limited to an infrared camera, an RGB camera, a ToF (Time of Flight) camera, a fish-eye camera, a light field camera, and an event camera.

[0057] According to the above concept, setting, and collection, collect information from different angles inside the vehicle corresponding to the preset intelligent cockpit to obtain corresponding in-vehicle condition information data. The in-vehicle condition information data includes but is not limited to in-vehicle occupant behavior information, light information, and sound information.

[0058] In the embodiment of the present invention, by collecting out-of-vehicle condition information data corresponding to the preset intelligent cockpit based on a preset first-category information collection method, or collecting in-vehicle condition information data corresponding to the preset intelligent cockpit based on a preset second-category information collection method, and then according to the out-of-vehicle environment information corresponding to the information from different angles of the out-of-vehicle condition information data and the in-vehicle environment information corresponding to the information from different angles of the in-vehicle condition information data, and integrating the above information from different angles, it is possible to predict the tendency state that the driver is about to have according to different in-vehicle and out-of-vehicle conditions where the intelligent cockpit is located, and according to the future tendency conditions, predict the potentially suitable convenient interaction methods for the driver. Then, adopt the convenient interaction method adapted to the tendency state to perform the human-vehicle interaction between the driver and the vehicle, realizing the intelligent self-adaptation of the human-vehicle interaction method between the driver and the vehicle based on the perception of the in-vehicle and out-of-vehicle environments. On the premise of ensuring the driving safety of the driver, adopt a convenient interaction method for the driver to realize the human-vehicle interaction between the driver and the vehicle, which can improve the convenience and reliability of the human-vehicle interaction, enhance the efficiency and effect of the human-vehicle interaction in intelligent driving under complex and changeable conditions, and enhance the intelligent experience of intelligent driving.

[0059] In one embodiment, please refer to Figure 4 , Figure 4 which is the schematic diagram of the second sub-process of the intelligent cockpit scenario simulation and prediction method provided by the embodiment of the present invention. As Figure 4As shown, in this embodiment, according to the trend state, a target preset vehicle-human interaction method adapted to the driver is predicted, and based on the target preset vehicle-human interaction method, vehicle-human interaction is performed with the driver, including: S41. Respond to the vehicle-human interaction request of the driver, and determine the preset correspondence between the preset trend state and the preset vehicle-human interaction method; S42. Determine the corresponding target preset vehicle-human interaction method according to the trend state and the preset correspondence; S43. Determine the target information expression method that the driver should adopt according to the target preset vehicle-human interaction method; S44. Prompt the target information expression method to the driver, and obtain the driver-side information expressed by the driver based on the target information expression method, so as to obtain the driver expression information transmitted by the driver; S45. Perform vehicle-human interaction with the driver according to the driver expression information.

[0060] Explanatorily, the correspondence between the trend state and the vehicle-human interaction method is preset in advance, that is, the preset correspondence between the preset trend state and the preset vehicle-human interaction method. The preset correspondence represents the preset vehicle-human interaction method to be adopted under a certain preset trend state. The preset correspondence includes but is not limited to the correspondence between the preset sound context state and the preset air gesture recognition interaction method, the correspondence between the preset driving control state and the preset voice interaction method, and the correspondence between the preset low-light environment state and the preset touch interaction method. Among them, the correspondence between the preset sound context state and the preset air gesture recognition interaction method means that in the preset sound context state, the driver and the vehicle adopt the preset air gesture recognition interaction method for vehicle-human interaction; the correspondence between the preset driving control state and the preset voice interaction method means that in the preset driving control state, the driver and the vehicle adopt the preset voice interaction method for vehicle-human interaction; the correspondence between the preset low-light environment state and the preset touch interaction method means that in the preset low-light environment state, the driver and the vehicle adopt the preset touch interaction method for vehicle-human interaction.

[0061] According to the above concept and settings, in response to the driver's vehicle-human interaction request, for example, the driver first initiates the vehicle-human interaction request through customized voice to wake up the vehicle-human interaction system, and determines the preset correspondence between the preset tendency state and the preset vehicle-human interaction method. The preset correspondence is as described above. Then, according to the tendency state and the preset correspondence, the corresponding target preset vehicle-human interaction method is determined. The target preset vehicle-human interaction method represents the target preset vehicle-human interaction method suitable to be adopted in the tendency state. For example, in the preset sound context state, the preset air gesture recognition interaction method is used as the target preset vehicle-human interaction method, and the driver and the vehicle use the preset air gesture recognition interaction method for vehicle-human interaction. In the preset driving control state, the preset voice interaction method is used as the target preset vehicle-human interaction method, and the driver and the vehicle use the preset voice interaction method for vehicle-human interaction. Then, according to the target preset vehicle-human interaction method, the target information expression method that the driver should adopt is determined. The target information expression method represents the way for the driver to express information. The target information expression method includes but is not limited to voice, gesture or touch, and the target information expression method is prompted to the driver. For example, the vehicle prompts the driver "Please speak out your needs" or "Please express your needs with gestures". Then the driver expresses the corresponding information based on the target information expression method, and the vehicle interaction system obtains the driver-side information expressed by the driver based on the target information expression method, and obtains the driver expression information transmitted by the driver. The driver expression information represents what the driver requires the interaction system of the vehicle to do. The vehicle interaction system analyzes and processes the driver expression information to interact with the driver for vehicle-human interaction.

[0062] Further, determining the corresponding target preset vehicle-human interaction method according to the tendency state and the preset correspondence includes at least one of the following: When the tendency state is the sound context state, determining the target preset vehicle-human interaction method as the preset air gesture recognition interaction method; When the tendency state is the driving control state, determining the target preset vehicle-human interaction method as the preset voice interaction method; When the tendency state is the low-light environment state, determining the target preset vehicle-human interaction method as the preset voice interaction method or the preset touch interaction method.

[0063] Specifically, when the tendency state is the voice context state, that is, in the case where there are interfering sounds corresponding to sounds including but not limited to conversation sounds, broadcast sounds, music sounds, etc. inside the vehicle, the target preset vehicle-human interaction method is determined to be the preset air gesture recognition interaction method, that is, the driver needs to use preset air gestures to interact with the vehicle to facilitate the driver's gesture actions. Similarly, when the tendency state is the driving control state, that is, in the case where the driver's hands need to be concentrated on the steering wheel for driving control, the target preset vehicle-human interaction method is determined to be the preset voice interaction method, that is, the driver needs to use voice to interact with the vehicle. When the tendency state is the low-light environment state, that is, in the case where the light inside the vehicle is not bright enough, the target preset vehicle-human interaction method is determined to be the preset voice interaction method or the preset touch interaction method, that is, the driver needs to use voice or touch to interact with the vehicle. Thus, according to different tendency situations, the potentially suitable convenient interaction method for the driver is predicted, and then the convenient interaction method adapted to the tendency state is adopted to perform the vehicle-human interaction between the driver and the vehicle, realizing the intelligent adaptation of the vehicle-human interaction method between the driver and the vehicle based on the perception of the vehicle interior and exterior environments. On the premise of ensuring the driving safety of the driver, the convenient interaction method of the driver is adopted to realize the vehicle-human interaction between the driver and the vehicle, which can improve the convenience and reliability of the vehicle-human interaction, enhance the vehicle-human interaction efficiency and effect of intelligent driving in complex and changeable situations, and enhance the intelligent experience of intelligent driving.

[0064] In an embodiment of the present invention, by responding to the vehicle-human interaction request of the driver, a preset correspondence between a preset trend state and a preset vehicle-human interaction method is determined, and according to the trend state and the preset correspondence, a corresponding target preset vehicle-human interaction method is determined. Then, according to the target preset vehicle-human interaction method, a target information expression method that the driver should adopt is determined. Next, the target information expression method is prompted to the driver, and the driver-side information expressed by the driver based on the target information expression method is obtained, and the driver expression information transmitted by the driver is obtained. Then, according to the driver expression information, vehicle-human interaction is carried out with the driver. It is possible to respond to the vehicle-human interaction request of the driver and prompt the predicted convenient interaction method suitable for the driver, that is, the target information expression method, to the driver, so as to carry out vehicle-human interaction between the driver and the vehicle based on the target information expression method. It can improve the matching accuracy of the intelligent cockpit scenario simulation and prediction and the complex and changeable conditions inside and outside the vehicle where the intelligent cockpit is located and the constantly changing conditions of the driver, thereby improving the accuracy of the intelligent cockpit scenario simulation and prediction, so that the vehicle-human interaction can better adapt to the complex and changeable vehicle conditions inside and outside the vehicle where the intelligent cockpit is located and the constantly changing personal conditions of the driver, realizing the intelligent self-adaptation of the vehicle-human interaction method between the driver and the vehicle based on the perception of the environment inside and outside the vehicle. On the premise of ensuring the driving safety of the driver, a convenient interaction method for the driver is adopted to realize the vehicle-human interaction between the driver and the vehicle, which can improve the convenience and reliability of the vehicle-human interaction, enhance the vehicle-human interaction efficiency and effect of intelligent driving in complex and changeable conditions, and enhance the intelligent experience of intelligent driving.

[0065] It should be noted that for the intelligent cockpit scenario simulation and prediction methods described in the above various embodiments, the technical features included in different embodiments can be recombined as needed to obtain the combined implementation schemes, but all are within the protection scope required by the present invention.

[0066] In one embodiment, an intelligent cockpit scenario simulation and prediction system is provided. The intelligent cockpit scenario simulation and prediction system corresponds one-to-one with the intelligent cockpit scenario simulation and prediction method in the above embodiment. Please refer to Figure 5 , Figure 5 which is a schematic block diagram of the intelligent cockpit scenario simulation and prediction system provided by the embodiment of the present invention. As Figure 5 shown, the intelligent cockpit scenario simulation and prediction system 50 includes a first monitoring module 51, a first acquisition module 52, a first simulation module 53, a first prediction module 54 and a first interaction module 55. The detailed descriptions of the above functional modules are as follows: The first monitoring module 51 is used to monitor whether there is a driver in the preset intelligent cockpit; The first acquisition module 52 is used to, if the above monitoring is yes, acquire the information data of the conditions inside and outside the vehicle corresponding to the preset intelligent cockpit; The first simulation module 53 is configured to simulate the in-vehicle and out-of-vehicle scenarios corresponding to the preset intelligent cockpit according to the in-vehicle and out-of-vehicle condition information data; The first prediction module 54 is configured to predict the trending state corresponding to the current state of the driver based on the in-vehicle and out-of-vehicle scenarios; The first interaction module 55 is configured to predict a target preset vehicle-human interaction method suitable for the driver according to the trending state, and perform vehicle-human interaction with the driver based on the target preset vehicle-human interaction method.

[0067] In one embodiment, the first prediction module 54 includes: The first acquisition sub-module is configured to acquire the personal information data of the driver; The construction sub-module is configured to construct a user portrait corresponding to the driver according to the personal information data and the in-vehicle and out-of-vehicle scenarios; The first collection sub-module is configured to collect the current state corresponding to the driver; The first prediction sub-module is configured to predict the upcoming consecutive state corresponding to the current state according to the user portrait and the current state, and based on a preset driving state prediction model, to obtain the trending state corresponding to the current state of the driver.

[0068] In one embodiment, the first acquisition sub-module includes at least one of the following: The second acquisition sub-module is configured to acquire the personal profile information data of the driver; The second collection sub-module is configured to collect the multi-modal personality factor information data of the driver, and determine the personal personality information data of the driver based on a preset multi-modal fusion personality model; The third collection sub-module is configured to collect the multi-modal emotion factor information data of the driver, and determine the personal emotion information data of the driver based on a preset multi-modal fusion emotion model; The third acquisition sub-module is configured to acquire the historical behavior information data of the driver.

[0069] In one embodiment, the first collection module 52 includes at least one of the following: The fourth collection sub-module is configured to collect the out-of-vehicle condition information data corresponding to the preset intelligent cockpit based on a preset first category information collection method; The fifth collection sub-module is configured to collect the in-vehicle condition information data corresponding to the preset intelligent cockpit based on a preset second category information collection method.

[0070] In one embodiment, the fourth collection sub-module includes at least one of the following: The sixth acquisition sub-module is used to acquire the vehicle exterior light information data corresponding to the preset intelligent cockpit based on a preset first light acquisition device; The seventh acquisition sub-module is used to acquire the vehicle exterior road condition information data corresponding to the preset intelligent cockpit based on a preset road condition acquisition device; The eighth acquisition sub-module is used to acquire the vehicle exterior weather information data corresponding to the preset intelligent cockpit based on a preset weather acquisition method.

[0071] In one embodiment, the fifth acquisition sub-module includes at least one of the following: The ninth acquisition sub-module is used to acquire the vehicle interior light information data corresponding to the preset intelligent cockpit based on a preset second light acquisition device; The tenth acquisition sub-module is used to acquire the vehicle interior audio information data corresponding to the preset intelligent cockpit based on a preset audio acquisition device; The eleventh acquisition sub-module is used to acquire the vehicle interior occupant behavior information data corresponding to the preset intelligent cockpit based on a preset image acquisition device.

[0072] In one embodiment, the first interaction module 55 includes: The first determination sub-unit is used to respond to the vehicle-person interaction request of the driver and determine the preset corresponding relationship between the preset tendency state and the preset vehicle-person interaction method; The second determination sub-unit is used to determine the corresponding target preset vehicle-person interaction method according to the tendency state and the preset corresponding relationship; The third determination sub-unit is used to determine the target information expression method that the driver should adopt according to the target preset vehicle-person interaction method; The fourth acquisition sub-module is used to prompt the target information expression method to the driver and acquire the driver-side information expressed by the driver based on the target information expression method, so as to obtain the driver expression information transmitted by the driver; The interaction sub-module is used to perform vehicle-person interaction with the driver according to the driver expression information.

[0073] In one embodiment, the second determination sub-unit includes at least one of the following: The fourth determination sub-unit is used to determine that the target preset vehicle-person interaction method is a preset air gesture recognition interaction method when the tendency state is a sound context state; The fifth determination sub-unit is used to determine that the target preset vehicle-person interaction method is a preset voice interaction method when the tendency state is a driving control state; A sixth determination subunit, configured to determine that the target preset vehicle-human interaction method is a preset voice interaction method or a preset touch interaction method when the trend state is a low-light environment state.

[0074] In one embodiment, the intelligent cockpit scenario simulation and prediction system 50 further includes: A continuous monitoring module, configured to, when it is detected that there is no driver in the preset intelligent cockpit, not collect the in-vehicle and out-of-vehicle condition information data corresponding to the preset intelligent cockpit, and continuously monitor whether there is a driver in the preset intelligent cockpit.

[0075] The embodiment of the present invention provides an intelligent cockpit scenario simulation and prediction system. By monitoring whether there is a driver in the preset intelligent cockpit, when it is detected that there is a driver in the preset intelligent cockpit, the in-vehicle and out-of-vehicle condition information data corresponding to the preset intelligent cockpit is collected, and based on the in-vehicle and out-of-vehicle condition information data, the in-vehicle and out-of-vehicle scenarios corresponding to the preset intelligent cockpit are simulated. Then, based on the in-vehicle and out-of-vehicle scenarios, the trend state corresponding to the current state of the driver is predicted, and according to the trend state, the target preset vehicle-human interaction method suitable for the driver is predicted. And based on the target preset vehicle-human interaction method, vehicle-human interaction is performed with the driver, so that it is possible to predict the trend state corresponding to the actions, operations, emotions, etc. that the driver will perform, including but not limited to, according to the different in-vehicle and out-of-vehicle conditions of the intelligent cockpit. And according to the future trend conditions, the potentially suitable convenient interaction method for the driver is predicted, and then the convenient interaction method adapted to the trend state is used to perform vehicle-human interaction between the driver and the vehicle, so as to achieve vehicle-human interaction between the driver and the vehicle in a convenient interaction method for the driver without affecting the states of the driver's actions, operations, emotions, etc. that will occur, improve the matching accuracy of the intelligent cockpit scenario simulation and prediction and the complex and changeable in-vehicle and out-of-vehicle conditions of the intelligent cockpit and the changing conditions of the driver, thereby improving the accuracy of the intelligent cockpit scenario simulation and prediction, realizing the intelligent adaptation of the vehicle-human interaction method between the driver and the vehicle based on in-vehicle and out-of-vehicle environment perception, automatically providing the driver with a personalized, customized, changing, and convenient intelligent adaptive vehicle-human interaction method suitable for the trend state, improving the vehicle-human interaction efficiency and effect of intelligent driving in complex and changeable conditions, improving the convenience and reliability of vehicle-human interaction on the premise of ensuring the driving safety of the driver, enhancing the intelligent experience of intelligent driving, and enhancing the core competitiveness of vehicle-human interaction in the era of intelligent driving.

[0076] For the specific limitations of the intelligent cockpit scenario simulation and prediction system, reference can be made to the limitations of the intelligent cockpit scenario simulation and prediction method in the above text, which will not be elaborated here. Each module in the above intelligent cockpit scenario simulation and prediction system can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0077] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device, and the instruction device implements the functions specified in one process Figure 1 one process or multiple processes and / or boxes Figure 1 specified in one box or multiple boxes.

[0078] In the embodiments of the present invention, the relevant data collection complies with the requirements of relevant laws and regulations, such as GDPR (General Data Protection Regulation of the European Union) or the information security standards of other countries and regions.

[0079] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A method for simulating and predicting intelligent cockpit scenarios, characterized in that: include: Monitor whether there is a driver in the preset smart cockpit; If the above monitoring result is yes, collect the vehicle interior and exterior condition information data corresponding to the preset smart cockpit; According to the vehicle interior and exterior condition information data, simulate the vehicle interior and exterior scenes corresponding to the preset smart cockpit; Based on the in-vehicle and out-of-vehicle scenarios, predict the trend state corresponding to the current state of the driver; According to the trend state, a target preset human-vehicle interaction mode suitable for the driver is predicted, and based on the target preset human-vehicle interaction mode, human-vehicle interaction is performed with the driver.

2. The intelligent cockpit scenario simulation and prediction method according to claim 1, characterized in that: Based on the in-vehicle and out-of-vehicle scenarios, predicting the trend state corresponding to the current state of the driver, including: Obtaining personal information data of the driver; Constructing a user profile corresponding to the driver based on the personal information data and the in-vehicle and out-of-vehicle scenarios; Collecting the current state of the driver; According to the user portrait and the current state, and based on a preset driving state prediction model, the upcoming subsequent state corresponding to the current state is predicted, and the trend state corresponding to the current state of the driver is obtained.

3. The intelligent cockpit scenario simulation and prediction method according to claim 2, characterized in that: Obtaining personal information data of the driver, including at least one of the following: Obtaining personal information data of the driver; Collecting multimodal personality factor information data of the driver, and determining personal personality information data of the driver based on a preset multimodal fusion personality model; Collecting multimodal emotional factor information data of the driver, and determining personal emotional information data of the driver based on a preset multimodal fusion emotional model; Acquire the driver's historical behavior information data.

4. The intelligent cockpit scenario simulation and prediction method according to any one of claims 1 to 3, characterized in that: Collecting vehicle interior and exterior status information data corresponding to the preset smart cockpit includes at least one of the following: Based on a preset first category information collection method, collecting vehicle exterior condition information data corresponding to the preset smart cockpit; Based on the preset second category information collection method, the in-vehicle condition information data corresponding to the preset smart cockpit is collected.

5. The intelligent cockpit scenario simulation and prediction method according to claim 4, characterized in that: Based on the preset first category information collection method, collecting vehicle exterior condition information data corresponding to the preset smart cockpit includes at least one of the following: Based on a preset first light collection device, collect the vehicle exterior light information data corresponding to the preset smart cockpit; Based on a preset road condition collection device, collect the vehicle external road condition information data corresponding to the preset smart cockpit; Based on a preset weather collection method, the outside weather information data corresponding to the preset smart cockpit is collected.

6. The intelligent cockpit scenario simulation and prediction method according to claim 4, characterized in that: Based on the preset second category information collection method, collecting the in-vehicle status information data corresponding to the preset smart cockpit includes at least one of the following: Based on a preset second light collection device, collecting the vehicle interior light information data corresponding to the preset smart cockpit; Based on a preset audio collection device, collect in-vehicle audio information data corresponding to the preset smart cockpit; Based on a preset image acquisition device, the in-vehicle occupant behavior information data corresponding to the preset smart cockpit is collected.

7. The intelligent cockpit scenario simulation and prediction method according to any one of claims 1 to 3, characterized in that: According to the trend state, predicting a target preset human-vehicle interaction mode suitable for the driver, and performing human-vehicle interaction with the driver based on the target preset human-vehicle interaction mode, including: In response to the driver's request for human-vehicle interaction, determining a preset corresponding relationship between a preset trend state and a preset human-vehicle interaction mode; Determining a corresponding target preset human-vehicle interaction mode according to the trend state and the preset corresponding relationship; According to the target preset human-vehicle interaction mode, determining the target information expression mode that the driver should adopt; Prompting the target information expression method to the driver, and acquiring the driver-side information expressed by the driver based on the target information expression method, to obtain the driver expression information transmitted by the driver; Performing human-vehicle interaction with the driver based on information expressed by the driver.

8. The intelligent cockpit scenario simulation and prediction method according to claim 7, characterized in that: Determining a corresponding target preset human-vehicle interaction mode according to the trend state and the preset corresponding relationship includes at least one of the following: When the trending state is a sound context state, determining that the target preset human-vehicle interaction mode is a preset air gesture recognition interaction mode; When the trend state is a driving control state, determining the target preset human-vehicle interaction mode to be a preset voice interaction mode; When the trend state is a dark light environment state, it is determined that the target preset human-vehicle interaction mode is a preset voice interaction mode or a preset touch interaction mode.

9. The intelligent cockpit scenario simulation and prediction method according to any one of claims 1 to 3, characterized in that: The method further comprises: When it is detected that there is no driver in the preset smart cockpit, the vehicle interior and exterior condition information data corresponding to the preset smart cockpit is not collected, and the preset smart cockpit is continuously monitored to see whether there is a driver.

10. An intelligent cockpit scenario simulation and prediction system, characterized in that: include: The first monitoring module is used to monitor whether there is a driver in the preset smart cockpit; A first collection module is used to collect vehicle interior and exterior status information data corresponding to the preset smart cockpit if the above monitoring result is yes; A first simulation module, used to simulate the vehicle interior and exterior scenarios corresponding to the preset smart cockpit according to the vehicle interior and exterior status information data; A first prediction module, used to predict the trend state corresponding to the current state of the driver based on the scene inside and outside the vehicle; The first interaction module is used to predict a target preset human-vehicle interaction mode suitable for the driver according to the trend state, and perform human-vehicle interaction with the driver based on the target preset human-vehicle interaction mode.

Images