Situation awareness enhanced interaction method, system and equipment based on intelligent driving system

Abstract

The invention relates to the technical field of data analysis, and discloses a situation awareness enhanced interaction method, system and device based on an intelligent driving system. Firstly, a perception information processing mode representing perception logic of a driver for at least one area in a current driving scene comprising the at least one area is acquired. And then, human factor data generated by interaction between the driver and the intelligent driving system is obtained, state prediction is performed based on the human factor data, and a comprehensive state prediction result of the driver is obtained. And finally, based on the perception information processing mode and the comprehensive state prediction result, obtaining an interested area of the driver in the at least one area, namely a situation perception result, and outputting an information prompt on the situation perception result. According to the embodiment, the response of the driver in the future state in the current driving scene can be predicted based on the experience-rich perception information processing mode of the driver, and appropriate interaction feedback is triggered through comprehensive analysis.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a situation awareness enhanced interaction method, system and device based on an intelligent driving system. Background Art

[0002] For autonomous driving or driving assistance systems, it is necessary not only to accurately perceive the state of the vehicle itself and the surrounding road environment, but also to monitor and understand the driver's physiological and psychological state. In the process of autonomous driving or assisted driving, the driver's situational awareness ability will be affected by many factors such as the degree of system automation, the complexity of the driving environment, and the driver's own state. Therefore, it is necessary to propose a situational awareness enhancement interaction method based on an intelligent driving system. Summary of the invention

[0003] The embodiments of this specification are intended to solve at least one of the technical problems in the related art to a certain extent. To this end, the embodiments of this specification propose a situation awareness enhancement interaction method, system and device based on an intelligent driving system.

[0004] The embodiment of this specification provides a situation awareness enhancement interaction method based on an intelligent driving system, the method comprising:

[0005] Acquiring a perception information processing mode of a driver in a current driving scene, wherein the current driving scene includes at least one area, and the perception information processing mode represents the driver's perception logic of the at least one area;

[0006] Acquire human factor data generated by the interaction between the driver and the intelligent driving system, and perform state prediction based on the human factor data to obtain a comprehensive state prediction result of the driver;

[0007] Based on the perception information processing mode and the comprehensive state prediction result, the driver's situation awareness result is obtained, and an information prompt is output on the situation awareness result, wherein the situation awareness result represents the area of ​​interest to the driver in the at least one area.

[0008] In one embodiment, obtaining the driver's situation awareness result based on the perception information processing mode and the comprehensive state prediction result includes:

[0009] When the comprehensive state prediction result meets the preset conditions, based on the perception information processing mode and the driver's eye movement data, the area that the driver will next gaze at is determined and used as the area of ​​interest to obtain the driver's situation awareness result.

[0010] In one embodiment, the step of obtaining the driver's perception information processing mode in the current driving scene includes:

[0011] Obtain the cognitive processing mode corresponding to the current driving scenario;

[0012] Eye movement data generated by the driver interacting with the intelligent driving system is obtained, and based on the eye movement data, the driver's perception information processing mode in the current driving scene is extracted from the cognitive processing mode.

[0013] In one embodiment, the cognitive processing mode corresponding to the driving scenario is determined by:

[0014] Acquire eye movement data generated by the driver in the driving scene interacting with the intelligent driving system, and determine a visual information pattern based on the eye movement data;

[0015] By fusing and extracting visual information patterns of driving scenes similar to the driving scene, a cognitive processing pattern corresponding to the driving scene is obtained.

[0016] In one embodiment, the human factor data includes at least two of physiological data, behavioral data, and emotional data; and the state prediction based on the human factor data to obtain the comprehensive state prediction result of the driver includes:

[0017] Performing state prediction based on at least two of the physiological data, the behavioral data, and the emotional data to obtain multi-dimensional prediction results;

[0018] A comprehensive prediction is performed based on the multi-dimensional prediction results to obtain a comprehensive state prediction result of the driver.

[0019] In one embodiment, the physiological data includes at least one of eye movement data, electroencephalogram signal data, and electromyography signal data; the behavioral data includes at least one of operation data and driving behavior analysis data; and the state prediction based on at least two of the physiological data, the behavioral data, and the emotional data includes at least two of the following:

[0020] Predicting based on at least one of the eye movement data, the EEG signal data, and the EMG signal data, obtaining at least one of the information load prediction result of the driver, the cognitive load state prediction result of the driver, and the fatigue state prediction result of the driver;

[0021] Performing prediction based on at least one of the operation data and the driving behavior analysis data to obtain at least one of the driver's operation load prediction result and the driver's distraction state prediction result;

[0022] A prediction is performed based on the emotional data and the physiological data to obtain a prediction result of the driver's emotional state.

[0023] The embodiment of this specification provides a situation awareness enhancement interaction system based on an intelligent driving system, the system comprising:

[0024] an information processing module, configured to obtain a driver's perception information processing mode in a current driving scene, wherein the current driving scene includes at least one area, and the perception information processing mode represents the driver's perception of the at least one area;

[0025] A state prediction module, used to perform state prediction based on human factor data to obtain a comprehensive state prediction result of the driver;

[0026] The information prompt module is used to obtain the driver's situation perception result based on the perception information processing mode and the comprehensive state prediction result, and output information prompts on the situation perception result.

[0027] An embodiment of the present specification provides a computer device, which includes: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors to enable the one or more processors to implement the steps of the method described in any of the above embodiments.

[0028] The embodiments of this specification provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0029] An embodiment of the present specification provides a computer program product, wherein the computer program product includes instructions, and when the instructions are executed by a processor of a computer device, the computer device is enabled to perform the steps of the method described in any one of the above embodiments.

[0030] In the above-mentioned implementation of the specification, the situational awareness of the driver is enhanced by constructing an adaptive interactive system. First, a perceptual information processing mode representing the driver's perceptual logic of at least one area in the current driving scene including at least one area is obtained. Then, the human factor data generated by the driver's interaction with the intelligent driving system is obtained, and the state is predicted based on the human factor data to obtain the driver's comprehensive state prediction result. Finally, based on the perceptual information processing mode and the comprehensive state prediction result, the area of ​​interest to the driver in at least one area, that is, the situational awareness result, is obtained, and information prompts are output on the situational awareness result. The above-mentioned implementation is based on the perceptual information processing mode of experienced drivers, which can predict the driver's reaction in the future state in the current driving scene, and trigger appropriate interactive feedback through comprehensive analysis.

[0031] During driving, the driver needs to focus on specific tasks, and timely and accurate acquisition of key situation information is the basis for ensuring correct judgment and response. The above implementation method processes and analyzes the dynamic interaction between people, vehicles, roads, and the environment in real time. Combining this information, it is possible to infer and generate appropriate prompts for the driver's current situation, thereby supporting the driver to make better decisions and improve traffic safety. It can help the driver complete driving tasks more efficiently and improve driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1a A schematic diagram of a flow chart of a situation awareness enhancement interaction method based on an intelligent driving system provided in an embodiment of this specification;

[0033] Figure 1b A schematic diagram of dividing driving scenarios into areas provided in an embodiment of this specification;

[0034] Figure 2 A schematic diagram of a process for determining a perceptual information processing mode provided in an embodiment of this specification;

[0035] Figure 3a A schematic diagram of a process for determining a cognitive processing mode provided for an embodiment of this specification;

[0036] Figure 3b A schematic diagram of determining cognitive processing modes provided for embodiments of this specification;

[0037] Figure 4a A schematic diagram of a process for determining a comprehensive state prediction result provided in an embodiment of this specification;

[0038] Figure 4b A schematic diagram of a prediction result of a comprehensive state provided in an embodiment of this specification;

[0039] Figure 5A schematic diagram of a situation awareness enhancement interaction system based on an intelligent driving system provided in an embodiment of this specification;

[0040] Figure 6 An internal structural diagram of a computer device provided for an embodiment of this specification. DETAILED DESCRIPTION

[0041] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0042] In the related art, three subsystems are used to enhance the pilot's situational awareness (SA): implicit expert knowledge mining subsystem, workload level prediction subsystem, and prompt information push subsystem. The main goal of these subsystems is to solve the problems encountered by pilots during flight. However, its framework and method can also be applied to the driver's situational awareness, and only need to be adjusted and optimized according to the specific needs of the driving scenario.

[0043] In the related technology, first, the constraint weights of the design criteria are determined by using the fuzzy hierarchical comprehensive analysis method. Then, the flight mission is simulated and the pilot's situational awareness level is measured. Through regression analysis, a functional relationship model between layout design and pilot situational awareness is established. Next, the objective function of the intelligent layout algorithm is constructed based on the model, and the layout design object is encoded. Finally, the particle swarm algorithm is used to iteratively optimize the layout design. This method improves the cockpit layout design system, making the cockpit layout scheme more reasonable and optimized.

[0044] In the related technology, firstly, information such as the driver's personality, preferences, experience and psychological quality is collected to generate a static ability score, and the driver's physiological, psychological and behavioral status is monitored regularly to generate a dynamic ability score. Next, the current driving environment is evaluated, and the driving environment score is obtained by integrating various factors. Then, the driving task is evaluated to determine the task type. Finally, based on the evaluation results, it is determined whether takeover is needed and the type of takeover is determined. The system includes a driver static evaluation module, a driver dynamic evaluation module, a driving environment evaluation module, a driving task evaluation module and a human-machine matching evaluation module. Through the collaborative work of these modules, the takeover strategy of the automatic driving assistance system can be optimized, and it can be more reasonably determined whether takeover control is needed.

[0045] However, the methods in the related art usually only focus on one aspect of the problem, and fail to systematically consider the impact of changes in various factors in the human-vehicle-road-environment (i.e., driver, vehicle, road, and environment) system on the driver's situational awareness. In addition, the methods in the related art also fail to enhance the driver's situational awareness ability according to the changes in the driver's state.

[0046] Based on this, the embodiment of this specification provides a situational awareness enhancement interaction method based on an intelligent driving system. By building an adaptive interactive system, the driver's situational awareness ability is enhanced. First, a perceptual information processing mode representing the driver's perceptual logic of at least one area in the current driving scene including at least one area is obtained. Then, the human factor data generated by the driver's interaction with the intelligent driving system is obtained, and the state is predicted based on the human factor data to obtain the driver's comprehensive state prediction result. Finally, based on the perceptual information processing mode and the comprehensive state prediction result, the area of ​​interest to the driver in at least one area, that is, the situational awareness result, is obtained, and information prompts are output on the situational awareness result. The above-mentioned embodiment is based on the perceptual information processing mode of experienced drivers, which can predict the driver's reaction in the future state in the current driving scene, and trigger appropriate interactive feedback through comprehensive analysis.

[0047] During driving, the driver needs to focus on specific tasks, and timely and accurate acquisition of key situation information is the basis for ensuring correct judgment and response. The above implementation method processes and analyzes the dynamic interaction between people, vehicles, roads, and the environment in real time. Combining this information, it is possible to infer and generate appropriate prompts for the driver's current situation, thereby supporting the driver to make better decisions and improve traffic safety. It can help the driver complete driving tasks more efficiently and improve driving safety.

[0048] This specification provides a situation awareness enhancement interaction method based on an intelligent driving system. Figure 1a , the method may include the following steps:

[0049] S110: Acquire the driver's perception information processing mode in the current driving scene.

[0050] S120: Obtain human factor data generated by the interaction between the driver and the intelligent driving system, and perform state prediction based on the human factor data to obtain a comprehensive state prediction result of the driver.

[0051] The current driving scene includes at least one area, the perception information processing mode

[0052] Specifically, according to the specific conditions of the current driving scene (such as road conditions, traffic conditions, weather, etc.), the perception logic of at least one area of ​​the driver in the current driving scene is obtained to obtain the perception information processing mode. Next, the interaction data between the driver and the intelligent driving system is obtained. These data include the driver's behavioral responses (such as braking, acceleration, steering, etc.) and the driver's physiological and psychological states (such as fatigue, anxiety, etc.). These human factors data reflect the driver's immediate reaction and emotional state, and can better understand the driver's current state. The human factors data are predicted by the algorithm, and the driver's emotions, reaction speed, concentration and other factors are comprehensively analyzed to predict the driver's overall state in the current scene, and obtain the driver's comprehensive state prediction result.

[0053] For example, see Figure 1b , Figure 1b The current driving scene in is divided into 9 areas.

[0054] S130: Based on the perception information processing mode and the comprehensive state prediction result, the situation perception result of the driver is obtained, and information prompts are output based on the situation perception result.

[0055] The situation awareness result indicates an area of ​​interest to the driver in at least one area.

[0056] Specifically, the perception information processing mode represents the order and logic of the driver's processing of the different divided areas in the current driving scene. After obtaining the driver's perception information processing mode and the comprehensive state prediction results, based on the comprehensive analysis of the two, the next area that the driver will look at in the current driving scene will be predicted and used as the driver's area of ​​interest. Then, information prompts will be output in the area of ​​interest to remind the driver to pay attention to possible risks or changes.

[0057] In the above implementation, an adaptive interactive system is constructed to enhance the situational awareness of the driver. First, a perceptual information processing mode representing the driver's perceptual logic of at least one area in the current driving scene including at least one area is obtained. Then, the human factor data generated by the driver's interaction with the intelligent driving system is obtained, and the state is predicted based on the human factor data to obtain the driver's comprehensive state prediction result. Finally, based on the perceptual information processing mode and the comprehensive state prediction result, the area of ​​interest to the driver in at least one area, that is, the situational awareness result, is obtained, and information prompts are output on the situational awareness result. The above implementation is based on the perceptual information processing mode of experienced drivers, which can predict the driver's reaction in the future state in the current driving scene, and trigger appropriate interactive feedback through comprehensive analysis.

[0058] During driving, the driver needs to focus on specific tasks, and timely and accurate acquisition of key situation information is the basis for ensuring correct judgment and response. The above implementation method processes and analyzes the dynamic interaction between people, vehicles, roads, and the environment in real time. Combining this information, it is possible to infer and generate appropriate prompts for the driver's current situation, thereby supporting the driver to make better decisions and improve traffic safety. It can help the driver complete driving tasks more efficiently and improve driving safety.

[0059] In some embodiments, based on the perception information processing mode and the comprehensive state prediction result, the driver's situational awareness result is obtained, including: when the comprehensive state prediction result meets the preset conditions, based on the perception information processing mode and the driver's eye movement data, determining the area that the driver will next gaze at and using the area as the area of ​​interest to obtain the driver's situational awareness result.

[0060] Specifically, under the premise that the comprehensive state prediction result meets the preset conditions (for example, when the comprehensive state prediction result is high), through real-time analysis of the driver's eye movement data (such as the eye's gaze point, gaze time and its change trajectory), combined with the driver's perception information processing mode, that is, the driver's attention and processing logic of external information during driving, it is inferred that the next area that the driver may be about to look at is used as the area of ​​interest. The area of ​​interest is the driver's situational awareness result.

[0061] In the above implementation, when the comprehensive state prediction result meets the preset conditions, the area where the driver will next be gazed is determined based on the perception information processing mode and the driver's eye movement data and the area is used as the area of ​​interest to obtain the driver's situational awareness result, which enables the driver to obtain key situational information in a timely and accurate manner, helping the driver to complete driving tasks more efficiently and improving driving safety.

[0062] In some embodiments, see Figure 2 , obtaining the driver's perception information processing mode in the current driving scene, may include the following steps:

[0063] S210: Obtain a cognitive processing mode corresponding to the current driving scenario.

[0064] S220: Acquire eye movement data generated by the interaction between the driver and the intelligent driving system, and extract the driver's perception information processing mode in the current driving scene from the cognitive processing mode based on the eye movement data.

[0065] Specifically, according to the specific conditions of the current driving scene (such as road conditions, traffic conditions, weather, etc.), the cognitive processing mode corresponding to the current driving scene is obtained. Next, during the interaction between the intelligent driving system and the driver, the driver's data is recorded and analyzed in real time. Through these data, the system can infer the driver's focus of attention at a certain moment, and further extract the driver's information processing mode in the current driving scene. Then, according to the determined information processing mode, the partial cognitive processing mode corresponding to the driver in the current driving scene is extracted from the cognitive processing mode to obtain the driver's perceptual information processing mode.

[0066] In the above implementation, the cognitive processing mode corresponding to the current driving scene is obtained, the eye movement data generated by the interaction between the driver and the intelligent driving system is obtained, and based on the eye movement data, the driver's perceptual information processing mode in the current driving scene is extracted from the cognitive processing mode, which helps to improve the coordination and interaction efficiency between the driver and the system.

[0067] In some embodiments, see Figure 3a , the cognitive processing mode corresponding to the driving scenario is determined by:

[0068] S310: Acquire eye movement data generated by the interaction between the driver and the intelligent driving system in the driving scene, and determine a visual information pattern based on the eye movement data.

[0069] Specifically, the visual information pattern represents the driver's attention distribution, information processing priority, and decision-making basis in a specific situation. By analyzing the driver's eye movement data, the system can deeply understand how the driver perceives and processes key information in the road environment in different situations, thereby simulating his decision-making process. In order to more accurately characterize this information processing process, the different information areas of the driving scene are first divided, and multiple different areas are set according to situational requirements. The division of multiple different areas is usually based on the importance of information in the road environment. By analyzing the eye movement data generated when the driver interacts with the intelligent driving system, the driver's attention trajectory between these areas can be determined. Finally, the driver's visual information pattern is constructed based on these trajectories, which reflects how he perceives and processes key information in the actual driving process.

[0070] S320: Obtain a cognitive processing pattern corresponding to the driving scene by fusing and extracting visual information patterns of driving scenes similar to the driving scene.

[0071] Specifically, the driver's cognitive processing mode refers to their inherent patterns in visual information processing when facing road situations, especially experienced drivers. These patterns help drivers understand the current situation and make predictions and decisions based on these perceptual information. For example, when a driver approaches an intersection, he or she usually gives priority to targets such as traffic lights, the movements of surrounding vehicles, and pedestrians. This information processing mode may vary between different drivers, but usually has certain similarities. The driving scenarios similar to the driving scenarios are determined by situational similarity. Situational similarity can be used to evaluate the similarity of different driving scenarios by scoring the same driving task by multiple experienced drivers. The visual information patterns in driving scenarios similar to the driving scenarios are integrated and extracted to construct a more general and accurate driver visual information model, and the cognitive processing patterns corresponding to the driving scenarios are obtained. The cognitive processing patterns help predict the driver's behavior and reaction in future driving scenarios, and further improve the intelligent driving system's understanding of the driver's cognitive state.

[0072] In some embodiments, multiple cognitive processing mode rule bases can be constructed according to different task scenario requirements. These rule bases serve as the basis for presenting prompt information, and the system can directly search the rule bases according to the current driving scenario to obtain relevant cognitive processing modes.

[0073] For example, for experienced drivers, based on the integration of domain knowledge (theoretical knowledge, such as traffic rules, judgment of road conditions, interpretation of driving information, etc.) and experience, they will have different eye movement trajectories for different scenes, thereby obtaining different visual information patterns. Figure 3b , for scene 1, obtain the sequence of personal interest regions of similar scenes. Then, obtain the visual information pattern based on the sequence of personal interest regions of similar scenes. By fusing and extracting the visual information patterns of driving scenes similar to the driving scenes, the cognitive processing pattern corresponding to the driving scenes is obtained. For scenes 2, ..., scene n, repeat the above operations to obtain the cognitive processing patterns corresponding to the respective scenes.

[0074] In the above implementation, eye movement data generated by the interaction between the driver and the intelligent driving system in the driving scene is obtained, and the visual information pattern is determined based on the eye movement data. By fusing and extracting the visual information patterns of driving scenes similar to the driving scenes, the cognitive processing pattern corresponding to the driving scene is obtained, thereby achieving more personalized and intelligent driving assistance.

[0075] In some embodiments, see Figure 4a The human factor data includes at least two of physiological data, behavioral data, and emotional data; performing state prediction based on the human factor data to obtain a comprehensive state prediction result of the driver may include the following steps:

[0076] S410: Perform state prediction based on at least two of the physiological data, the behavioral data, and the emotional data to obtain multi-dimensional prediction results.

[0077] S420: Perform comprehensive prediction based on the multi-dimensional prediction results to obtain a comprehensive state prediction result of the driver.

[0078] Specifically, data collection using sensors, cameras and other equipment can obtain physiological data, behavioral data and emotional data. Data analysis technology is used to perform multi-dimensional predictive analysis on the driver's future state based on at least two of the physiological data, behavioral data and emotional data to obtain multi-dimensional prediction results. Then, the prediction results of multiple dimensions are integrated through a comprehensive algorithm, such as using weighted average, decision tree or deep learning model to obtain the driver's comprehensive state prediction result. The comprehensive state prediction result can help judge the driver's overall driving ability and safety risks, thereby providing real-time warnings in different situations (such as fatigue driving, emotional fluctuations, distraction, etc.). Among them, physiological data may include physiological indicators such as the driver's heart rate and skin galvanic response. Behavioral data may include the driver's behavior patterns such as speed changes and steering wheel rotation. Emotional data may include facial expression signals, arousal level, etc.

[0079] It should be noted that in order to make a more accurate comprehensive state prediction, a label setting based on subjective reports can be used. Specifically, the labels of subjective reports are divided into three levels: high, medium, and low, and the evaluation criteria are 10-level scores: low (1-3 points), medium (4-7 points), and high (8-10 points). The setting of these labels is based on the judgment of the driver's state by three blind evaluations (driver's own feedback, experienced driver evaluation, and experimenter observation). According to these evaluation results, the driver's current state is marked as high, medium, and low, and classified predictions are made based on this label.

[0080] For example, a logistic regression model is used to perform comprehensive prediction using multi-dimensional prediction results to obtain a comprehensive prediction result of the driver's state. Figure 4b The multi-dimensional prediction results may include information load, operation load, cognitive load, fatigue state, emotional state and distraction state. Based on the first time interval, the information load X of the interval is obtained. 1-1 , Operation load X 2-1 , cognitive load X 3-1 , Fatigue status X 4-1 , emotional state X 5-1 and distraction state X 6-1 Based on the second time interval, obtain the information load X of the interval 1-2 , Operation load X 2-2 , cognitive load X3-2 , Fatigue status X 4-2 , emotional state X 5-2 and distraction state X 6-2 Based on the third time interval, the information load X of the interval is obtained. 1-3 , Operation load X 2-3 , cognitive load X 3-3 , Fatigue status X 4-3 , emotional state X 5-3 and distraction state X 6-3 By analyzing the prediction result of the third time interval, the prediction result of the comprehensive state of the driver in the fourth time interval can be predicted.

[0081] In the above embodiment, state prediction is performed based on at least two of the physiological data, behavioral data and emotional data to obtain a multi-dimensional prediction result, and a comprehensive prediction is performed based on the multi-dimensional prediction result to obtain a comprehensive state prediction result of the driver, which can more accurately evaluate the predicted state of the driver.

[0082] In some embodiments, the physiological data includes at least one of eye movement data, electroencephalogram signal data, and electromyography signal data; the behavioral data includes at least one of operation data and driving behavior analysis data; and the state prediction based on at least two of the physiological data, behavioral data, and emotional data includes at least two of the following:

[0083] Based on at least one of the eye movement data, the electroencephalogram signal data, and the electromyography signal data, a prediction is performed to obtain at least one of the driver's information load prediction result, the driver's cognitive load state prediction result, and the driver's fatigue state prediction result.

[0084] Prediction is performed based on at least one of the operation data and the driving behavior analysis data to obtain at least one of a driver's operation load prediction result and a driver's distraction state prediction result.

[0085] The prediction result of the driver's emotional state is obtained based on the emotional data and physiological data.

[0086] Specifically, the driver's information load is mainly affected by the perceived information, especially the processing of visual information. By analyzing the eye movement data, especially the number of eye movement shifts per minute (i.e., the frequency of eye movement shifts from one focus to another), the information load that the driver is subjected to during driving can be effectively evaluated, and the driver's information load prediction results can be obtained. A higher number of eye movement shifts usually means that the driver needs to frequently process a variety of visual information, thereby increasing the information load; conversely, a lower number of shifts may indicate that the driver's attention is focused on a small amount of important information.

[0087] The driver's cognitive load state during driving is affected by many factors, including information load (such as visual information and auditory information in the driving environment) and operation load (such as the complexity of controlling the vehicle, the frequency of operation, etc.). The combined effect of these factors leads to changes in the driver's mental load level. In order to evaluate the driver's cognitive load state, the driver's EEG (electroencephalogram) signal can be monitored in real time, and the key indicators of the EEG signal data, such as the frequency distribution and amplitude change of brain waves, can be analyzed. The EEG signal data per minute can reflect the driver's mental state and cognitive load level in different driving scenarios. For example, higher alpha waves and theta waves are usually associated with low cognitive load, while higher beta waves may indicate higher cognitive load. By analyzing these EEG signals, the driver's cognitive load state prediction results can be obtained.

[0088] During long driving, the driver will gradually feel tired due to continuous operation and concentration, especially cumulative fatigue will affect their reaction speed and judgment ability. In order to evaluate the driver's fatigue state, it can be predicted by real-time monitoring of the driver's electromyographic signal (EMG) changes. The electromyographic signal reflects the muscle activity of the driver during the operation, especially the muscles in the arms, shoulders, neck and other parts related to driving operations. By analyzing the change trend of the electromyographic signal data per minute, the driver's fatigue state prediction result can be obtained. For example, cumulative fatigue is usually accompanied by abnormal patterns of muscle activity, such as frequent micro-contractions or maintaining a fixed posture for a long time, which will lead to the gradual accumulation of muscle fatigue.

[0089] Drivers need to continuously perform a variety of interactive operations during driving, such as adjusting the steering wheel, operating pedals, adjusting the vehicle entertainment system, using navigation, and interacting with other traffic participants. These operations will directly affect the driver's workload, especially in complex driving environments. The driver's operating load will increase with the increase in the frequency of operations. In order to evaluate the workload caused by interactive operations, the number of operations performed by the driver per minute can be monitored to obtain operation data to measure its information load. The change in the number of operations per minute reflects the driver's work intensity and information processing needs in a specific situation. A higher frequency of operations may indicate that the driver needs more judgment and reaction when facing a complex environment, resulting in an increase in cognitive load. The operation data is analyzed and predicted to obtain the prediction results of the driver's operation load.

[0090] During driving, the driver's distraction state may be affected by many factors, including changes in the external environment, internal task burden, and the driver's attention allocation. When the driver's attention is not focused, his control of the vehicle may become imprecise, resulting in abnormal changes in vehicle behavior. Therefore, the evaluation of the driver's distraction level can be achieved by analyzing the vehicle behavior data. By acquiring the vehicle behavior data, the driving behavior analysis data is obtained. The driving behavior analysis data is analyzed and predicted to obtain the driver's distraction state prediction result.

[0091] Exemplarily, the driving behavior analysis data may be at least one of the lateral acceleration and steering wheel angle data of the vehicle. The lateral acceleration (e.g., lateral movement of the vehicle) and the steering wheel angle data are important indicators reflecting the driver's operating accuracy. When the driver is distracted, the lateral acceleration and steering wheel angle of the vehicle may fluctuate greatly. For example, when distracted, the driver may fail to adjust the steering wheel or brake in time, resulting in irregular deviations in the vehicle's driving trajectory or unnecessary direction corrections. The amplitude and frequency of these fluctuations can be used as a basis for assessing the driver's distraction state. For example, a certain threshold can be set to analyze in real time through vehicle behavior data whether the driver exceeds the normal operating range. When the lateral acceleration exceeds the set threshold or the steering wheel angle changes by more than a certain range, it may indicate that the driver has cognitive distraction at a certain moment and cannot accurately control the vehicle.

[0092] During driving, the driver's emotional changes have an important impact on driving behavior, especially when the driver's emotions fluctuate greatly, which may affect his judgment and reaction ability to road conditions. In order to accurately evaluate the driver's emotional state, multi-dimensional analysis can be performed by combining physiological data and emotional data (such as facial expression signals and emotional valence), and the driver's emotional fluctuations and arousal degree can be monitored and predicted in real time to obtain the driver's emotional state prediction results. Among them, physiological data includes at least one of eye movement data, electroencephalogram signal data, and electromyography signal data.

[0093] Among them, emotional valence is usually expressed in the range of [-1, 1], where -1 represents extremely negative emotions, 1 represents extremely positive emotions, and 0 represents neutral emotions. In order to quantify the driver's emotional state, features related to emotional changes can be extracted by analyzing physiological information (such as heart rate, skin galvanic response, breathing rate, etc.) and facial expression signals (such as facial muscle activity, micro-expressions, eye movements, etc.) every minute. These physiological and facial expression features can reflect the driver's emotional valence and arousal level. For example, when a driver experiences negative emotions such as excitement, anxiety, or anger, he or she will show a higher heart rate, a faster breathing rate, or a tense facial expression; on the contrary, a positive emotional state (such as pleasure or relaxation) may be accompanied by a lower physiological activation and a more relaxed facial expression.

[0094] For example, in order to accurately predict the driver's state, all the above physiological and behavioral data can be classified and evaluated by machine learning algorithms (such as KNN, support vector machine, etc.). By comprehensively analyzing the eye movement data, EEG signals, EMG signals, operation data, vehicle behavior data and emotional data every minute, the driver's cognitive load, fatigue, distraction and emotional state can be predicted in real time.

[0095] In the above implementation, prediction is performed based on at least one of the eye movement data, the EEG signal data, and the electromyography signal data to obtain at least one of the driver's information load prediction result, the driver's cognitive load state prediction result, and the driver's fatigue state prediction result; prediction is performed based on at least one of the operation data and the driving behavior analysis data to obtain at least one of the driver's operation load prediction result and the driver's distraction state prediction result; prediction is performed based on the emotional data and the physiological data to obtain the driver's emotional state prediction result, thereby providing a data basis for the subsequent determination of the comprehensive state prediction result.

[0096] This specification provides a situation awareness enhancement interaction system 500 based on an intelligent driving system. Figure 5 The situation awareness enhancement interaction system 500 based on the intelligent driving system includes: an information processing module 510, a state prediction module 520, and an information prompt module 530.

[0097] An information processing module 510, configured to obtain a driver's perception information processing mode in a current driving scene, wherein the current driving scene includes at least one area, and the perception information processing mode represents the driver's perception of the at least one area;

[0098] A state prediction module 520, configured to perform state prediction based on human factor data to obtain a comprehensive state prediction result of the driver;

[0099] The information prompt module 530 is used to obtain the driver's situation perception result based on the perception information processing mode and the comprehensive state prediction result, and output information prompts based on the situation perception result.

[0100] For a specific description of the situational awareness enhanced interaction system based on the intelligent driving system, please refer to the description of the situational awareness enhanced interaction method based on the intelligent driving system above, which will not be repeated here.

[0101] An embodiment of the present specification provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method steps in the above embodiment are implemented.

[0102] An embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0103] One embodiment of the present specification provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method of any of the above embodiments.

[0104] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a situational awareness enhancement interaction method based on an intelligent driving system is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse.

[0105] Those skilled in the art will understand that Figure 6The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution disclosed in this specification, and does not constitute a limitation on the computer device to which the solution disclosed in this specification is applied. Specifically, the computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0106] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

Claims

1. A situation awareness enhancement interaction method based on an intelligent driving system, characterized in that: The method comprises: Acquiring a perception information processing mode of a driver in a current driving scene, wherein the current driving scene includes at least one area, and the perception information processing mode represents the driver's perception logic of the at least one area; Acquire human factor data generated by the interaction between the driver and the intelligent driving system, and perform state prediction based on the human factor data to obtain a comprehensive state prediction result of the driver; Based on the perception information processing mode and the comprehensive state prediction result, the driver's situation awareness result is obtained, and an information prompt is output on the situation awareness result, wherein the situation awareness result represents the area of ​​interest to the driver in the at least one area.

2. The method according to claim 1, characterized in that The step of obtaining the driver's situation awareness result based on the perception information processing mode and the comprehensive state prediction result includes: When the comprehensive state prediction result meets the preset conditions, based on the perception information processing mode and the driver's eye movement data, the area that the driver will next gaze at is determined and used as the area of ​​interest to obtain the driver's situation awareness result.

3. The method according to claim 1, characterized in that The obtaining of the driver's perception information processing mode in the current driving scene includes: Obtain the cognitive processing mode corresponding to the current driving scenario; Eye movement data generated by the driver interacting with the intelligent driving system is obtained, and based on the eye movement data, the driver's perception information processing mode in the current driving scene is extracted from the cognitive processing mode.

4. The method according to claim 3, characterized in that Determine the cognitive processing mode corresponding to the driving scenario by: Acquire eye movement data generated by the driver in the driving scene interacting with the intelligent driving system, and determine a visual information pattern based on the eye movement data; By fusing and extracting visual information patterns of driving scenes similar to the driving scene, a cognitive processing pattern corresponding to the driving scene is obtained.

5. The method according to claim 1, characterized in that The human factor data includes at least two of physiological data, behavioral data and emotional data; the state prediction based on the human factor data to obtain the comprehensive state prediction result of the driver includes: Performing state prediction based on at least two of the physiological data, the behavioral data, and the emotional data to obtain multi-dimensional prediction results; A comprehensive prediction is performed based on the multi-dimensional prediction results to obtain a comprehensive state prediction result of the driver.

6. The method according to claim 5, characterized in that The physiological data includes at least one of eye movement data, electroencephalogram signal data, and electromyography signal data; the behavioral data includes at least one of operation data and driving behavior analysis data; and the state prediction based on at least two of the physiological data, the behavioral data, and the emotional data includes at least two of the following: Predicting based on at least one of the eye movement data, the EEG signal data, and the EMG signal data, obtaining at least one of the information load prediction result of the driver, the cognitive load state prediction result of the driver, and the fatigue state prediction result of the driver; Performing prediction based on at least one of the operation data and the driving behavior analysis data to obtain at least one of the driver's operation load prediction result and the driver's distraction state prediction result; A prediction is performed based on the emotional data and the physiological data to obtain a prediction result of the driver's emotional state.

7. A situation awareness enhancement interactive system based on an intelligent driving system, characterized in that: The system comprises: an information processing module, configured to obtain a driver's perception information processing mode in a current driving scene, wherein the current driving scene includes at least one area, and the perception information processing mode represents the driver's perception of the at least one area; A state prediction module, used to perform state prediction based on human factor data to obtain a comprehensive state prediction result of the driver; The information prompt module is used to obtain the driver's situation perception result based on the perception information processing mode and the comprehensive state prediction result, and output information prompts on the situation perception result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images