Brain-eye fusion measurement system for traffic participant risk perception based on simulation platform

Abstract

The invention provides a brain-eye fusion measurement system for traffic participant risk perception based on a simulation platform, and relates to the technical field of human factors, and the system comprises the simulation platform which is used for carrying out traffic scene simulation interaction with traffic participants; the questionnaire data acquisition module is used for acquiring risk perception questionnaire data filled by traffic participants; the physiological data acquisition module is used for acquiring physiological data when the traffic participant interacts with the simulation platform; the physiological index calculation module is used for calculating physiological indexes based on the collected data, and the physiological indexes comprise an electroencephalogram index, an eye movement index and an emotion index; and the risk perception measurement module is used for calculating a risk perception reference value based on the risk perception questionnaire data, and constructing a risk perception classification measurement model by taking the risk perception reference value as a risk perception classification label and taking the physiological indexes as features. With the adoption of the scheme, the risk perception measurement of the road traffic participants is realized.

Description

Technical Field

[0001] The present application relates to the technical field of ergonomics, and in particular to a brain-eye fusion measurement system method and device for risk perception of traffic participants based on a simulation platform. Background Art

[0002] Traffic accidents have become one of the major reasons threatening human life and property. Understanding the risk perception and decision-making process of road traffic users will help prevent the occurrence of unsafe road traffic behaviors from the root. Human-vehicle interaction experiments in real-life scenarios are constrained by laws, regulations, ethics, safety and other conditions, making them difficult to carry out. At the same time, risk perception quantification schemes based on interviews, questionnaires and other survey forms also have many limitations such as time lag and subjective bias. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first purpose of this application is to propose a brain-eye fusion measurement system for risk perception of traffic participants based on a simulation platform, which solves the technical problems that existing methods are difficult to achieve and have great limitations, and realizes the risk perception measurement of road traffic participants.

[0005] The second objective of this application is to propose a brain-eye fusion measurement method for risk perception of traffic participants based on a simulation platform.

[0006] The third objective of the present application is to provide a computer device.

[0007] A fourth object of the present application is to provide a non-transitory computer-readable storage medium.

[0008] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a brain-eye fusion measurement system for risk perception of traffic participants based on a simulation platform, comprising:

[0009] The simulation platform is based on the simulated road traffic scene, uses an external display as a visual interaction medium, and uses a keyboard as a behavioral interaction medium to conduct traffic scene simulation interaction with traffic participants;

[0010] Questionnaire data collection module, used to obtain risk perception questionnaire data filled out by traffic participants;

[0011] Physiological data collection module, used to collect EEG data, eye movement data, facial expression data and behavioral data when traffic participants interact with the simulation platform;

[0012] A physiological index calculation module, used to calculate physiological indexes based on the collected data, wherein the physiological indexes include EEG indexes, eye movement indexes and emotion indexes;

[0013] The risk perception measurement module is used to calculate the risk perception benchmark value based on the risk perception questionnaire data, use the risk perception benchmark value as the classification label of risk perception, and use physiological indicators as features to build a classification measurement model for risk perception.

[0014] Optionally, in one embodiment of the present application, the risk perception measurement module is also used to establish a structural equation model of risk perception-decision-making based on environmental variables, prior knowledge variables, unsafe behavior variables and perception degree variables, wherein the environmental variables are determined by traffic scene information, the prior knowledge variables are determined by risk perception questionnaire data filled out by traffic participants, the unsafe behavior variables are determined by the behavior data of traffic participants, and the perception degree variables represent the degree to which the user perceives the surrounding environment information, and the perception degree variables are determined by physiological indicators.

[0015] Optionally, in one embodiment of the present application, the physiological data acquisition module is specifically used to:

[0016] Use a portable EEG device to collect EEG data;

[0017] Use a screen-based eye tracker to collect eye movement data;

[0018] Use the camera to collect expression data;

[0019] The keyboard key operation data of the traffic participant is obtained, and the behavior data of the traffic participant is determined based on the keyboard key operation data.

[0020] Optionally, in one embodiment of the present application, the EEG index includes attention, vigilance, mental workload and mental fatigue level calculated based on time-frequency analysis and sub-band energy;

[0021] Eye movement metrics include fixation duration, fixation point coordinates, saccade velocity, saccade amplitude, and blink frequency extracted based on eye tracking;

[0022] Generate sentiment indicators, including:

[0023] Use computer vision algorithms to detect facial data and extract the position, distance, and angle changes of facial key points frame by frame;

[0024] Analyze facial muscle movements and detect skin texture and micro-expression changes using local binary patterns;

[0025] Extract and analyze action units associated with specific emotions using the Facial Action Coding System;

[0026] The activation degree, duration and frequency of basic emotional valence are calculated based on the position, distance and angle changes of facial key points, skin texture and micro-expression changes and action units.

[0027] To achieve the above-mentioned purpose, the second embodiment of the present invention proposes a brain-eye fusion measurement method for risk perception of traffic participants based on a simulation platform, comprising:

[0028] Obtain risk perception questionnaire data filled out by traffic participants;

[0029] Collect EEG data, eye movement data, facial expression data and behavioral data of traffic participants when they interact with the above simulation platform;

[0030] Calculate physiological indicators based on the collected data, where the physiological indicators include EEG indicators, eye movement indicators and emotional indicators;

[0031] Based on the risk perception questionnaire data, the risk perception benchmark value is calculated, and the risk perception benchmark value is used as the classification label of risk perception. Physiological indicators are used as features to construct a classification measurement model of risk perception.

[0032] Optionally, in one embodiment of the present application, a structural equation model of risk perception-decision-making is established based on environmental variables, prior knowledge variables, unsafe behavior variables and perception degree variables, wherein the environmental variables are determined by traffic scene information, the prior knowledge variables are determined by risk perception questionnaire data filled out by traffic participants, the unsafe behavior variables are determined by the behavior data of traffic participants, and the perception degree variable represents the degree to which the user perceives the surrounding environment information, and the perception degree variable is determined by physiological indicators.

[0033] Optionally, in one embodiment of the present application, the EEG data, eye movement data, expression data and behavior data of the traffic participants when interacting with the autonomous driving simulation platform are collected, including:

[0034] Use a portable EEG device to collect EEG data;

[0035] Use a screen-based eye tracker to collect eye movement data;

[0036] Use the camera to collect expression data;

[0037] The keyboard key operation data of the traffic participant is obtained, and the behavior data of the traffic participant is determined based on the keyboard key operation data.

[0038] Optionally, in one embodiment of the present application, the EEG index includes attention, vigilance, mental workload and mental fatigue level calculated based on time-frequency analysis and sub-band energy;

[0039] Eye movement metrics include fixation duration, fixation point coordinates, saccade velocity, saccade amplitude, and blink frequency extracted based on eye tracking;

[0040] Generate sentiment indicators, including:

[0041] Use computer vision algorithms to detect facial data and extract the position, distance, and angle changes of facial key points frame by frame;

[0042] Analyze facial muscle movements and detect skin texture and micro-expression changes using local binary patterns;

[0043] Extract and analyze action units associated with specific emotions using the Facial Action Coding System;

[0044] The activation degree, duration and frequency of basic emotional valence are calculated based on the position, distance and angle changes of facial key points, skin texture and micro-expression changes and action units.

[0045] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the brain-eye fusion measurement method of risk perception of traffic participants based on the simulation platform is implemented.

[0046] In order to achieve the above-mentioned objectives, the fourth aspect embodiment of the present invention proposes a non-temporary computer-readable storage medium, which, when the instructions in the storage medium are executed by a processor, enables the execution of the above-mentioned brain-eye fusion measurement method of traffic participant risk perception based on a simulation platform.

[0047] The brain-eye fusion measurement method of risk perception of traffic participants based on a simulation platform in the embodiment of the present application is based on a simulation platform experiment of a simulated scenario and measures risk perception indicators with the help of physiological indicators with higher time resolution and objectivity. It is a highly potential and feasible solution for understanding the traffic behavior of road users.

[0048] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0050] Figure 1 A schematic diagram of the structure of a brain-eye fusion measurement system for risk perception of traffic participants based on a simulation platform provided in Example 1 of the present application;

[0051] Figure 2 This is an example diagram of the 4-level wavelet packet decomposition process of an embodiment of the present application;

[0052] Figure 3A schematic diagram of the structure of a brain-eye fusion measurement system for risk perception of traffic participants based on the autonomous driving simulation platform carla in an embodiment of the present application;

[0053] Figure 4 This is a structural example diagram of the data collection function of an embodiment of the present application;

[0054] Figure 5 A flowchart of a method for measuring the risk perception of traffic participants by brain-eye fusion based on a simulation platform is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0056] The following describes the brain-eye fusion measurement system and device for risk perception of traffic participants based on a simulation platform according to an embodiment of the present application with reference to the accompanying drawings.

[0057] Figure 1 A schematic diagram of the structure of a brain-eye fusion measurement system for risk perception of traffic participants based on a simulation platform provided in Example 1 of the present application.

[0058] like Figure 1 As shown in the figure, the brain-eye fusion measurement system for traffic participant risk perception based on the simulation platform includes:

[0059] The simulation platform is based on the simulated road traffic scene, uses an external display as a visual interaction medium, and uses a keyboard as a behavioral interaction medium to conduct traffic scene simulation interaction with traffic participants;

[0060] Questionnaire data collection module, used to obtain risk perception questionnaire data filled out by traffic participants;

[0061] In this embodiment, the risk perception questionnaire data includes basic information, and the basic information includes personality attributes, risk seeking tendency, traffic history and experience, etc.

[0062] The data collection module is used to collect EEG data, eye movement data, expression data and behavior data of traffic participants when they interact with the simulation platform;

[0063] A physiological index calculation module, used to calculate physiological indexes based on the collected data, wherein the physiological indexes include EEG indexes, eye movement indexes and emotion indexes;

[0064] The risk perception measurement module is used to calculate the risk perception benchmark value based on the risk perception questionnaire data, use the risk perception benchmark value as the classification label of risk perception, and use physiological indicators as features to build a classification measurement model for risk perception.

[0065] In this embodiment, a risk perception measurement model is established to allow for quantitative measurement of participants' real-time risk perception levels during interactive behaviors by collecting participants' EEG, eye movement, and facial expression data in real time under experimental conditions, so as to further support theoretical research on unsafe behaviors.

[0066] The brain-eye fusion measurement method of risk perception of traffic participants based on a simulation platform in the embodiment of the present application is based on a simulation platform experiment of a simulated scenario and measures risk perception indicators with the help of physiological indicators with higher time resolution and objectivity. It is a highly potential and feasible solution for understanding the traffic behavior of road users.

[0067] Optionally, in one embodiment of the present application, the risk perception measurement module is also used to establish a structural equation model of risk perception-decision-making based on environmental variables, prior knowledge variables, unsafe behavior variables and perception degree variables, wherein the environmental variables are determined by traffic scene information, the prior knowledge variables are determined by risk perception questionnaire data filled out by traffic participants, the unsafe behavior variables are determined by the behavior data of traffic participants, and the perception degree variables represent the degree to which the user perceives the surrounding environment information, and the perception degree variables are determined by physiological indicators.

[0068] In this embodiment, based on the constructed risk perception-decision-making structural equation model, an unsafe behavior mechanism is established, including:

[0069] The unsafe behavior process of traffic participants, namely the risk perception-decision-making process, is a multi-level dynamic process involving the perception, understanding and prediction of environmental information. In traffic scenarios, traffic participants, under the guidance of specific environmental factors and individual prior knowledge, constantly perceive, understand and predict the surrounding traffic conditions, and make decisions on how to pass safely. Among the above factors, factors as environmental variables can be directly observed in real scenarios, such as environmental visibility, traffic flow, etc. Variables as individual prior knowledge can be obtained through questionnaire surveys to obtain personalized characteristics related to traffic behavior, such as personality traits, traffic experience, etc. The unsafe behavior variables of traffic participants can be obtained by observing and evaluating their behavior, such as whether they follow the instructions of traffic lights and whether they fail to comply with the requirements of road markings.

[0070] However, it is often difficult to directly measure the effectiveness of traffic participants' perception of surrounding environmental information, their interpretation and understanding of perceived information, and their predictions and response decisions based on this interpretation and understanding of future situations. This embodiment can use physiological sensing equipment to extract indicators from physiological data as observation variables of potential variables in the perception, understanding, and prediction stages, and combine them with the above-mentioned other variables to establish a structural equation model of risk perception-decision-making of traffic participants, and evaluate the fit of the structural equation model through the fitting index. According to the path coefficient, the degree of influence of different factors on risk perception and unsafe behavior can be interpreted, and how traffic participants perceive risks and make decisions in specific situations can be understood, thereby further clarifying the mechanism of their unsafe behavior.

[0071] Optionally, in one embodiment of the present application, the physiological data acquisition module is specifically used to:

[0072] Use a portable EEG device to collect EEG data;

[0073] Use a screen-based eye tracker to collect eye movement data;

[0074] Use the camera to collect expression data;

[0075] The keyboard key operation data of the traffic participant is obtained, and the behavior data of the traffic participant is determined based on the keyboard key operation data.

[0076] Optionally, in one embodiment of the present application, the EEG index includes attention, vigilance, mental workload and mental fatigue level calculated based on time-frequency analysis and sub-band energy, and the specific extraction process includes:

[0077] (1) Intercepting EEG signals during the task period based on behavioral labels;

[0078] (2) Perform 1-40 Hz bandpass filtering and 50 Hz notch filtering to filter out power frequency interference and intercept the frequency band of interest;

[0079] (3) Perform electrode average re-reference;

[0080] (4) Conduct independent component analysis and remove artifact components based on experience;

[0081] (5) Perform 4-level wavelet packet decomposition and divide the subbands into subbands. The structure is as follows: Figure 2 As shown;

[0082] (6) Retain the wavelet packet nodes of 1-40 Hz and calculate the subband energy based on the wavelet packet coefficients;

[0083] (7) Calculate the corresponding EEG index based on the sub-band energy;

[0084] Eye movement metrics included manually annotated regions of interest and event-based fixation duration, fixation point coordinates, saccade velocity, saccade amplitude, and blink frequency;

[0085] Generate sentiment indicators, including:

[0086] Use computer vision algorithms to detect facial data and extract the position, distance, and angle changes of facial key points frame by frame;

[0087] Analyze facial muscle movements and detect skin texture and micro-expression changes using local binary patterns;

[0088] Extract and analyze action units associated with specific emotions using the Facial Action Coding System;

[0089] The activation degree, duration and frequency of basic emotional valence are calculated based on the position, distance and angle changes of facial key points, skin texture and micro-expression changes and action units.

[0090] The embodiment of the present application also proposes a brain-eye fusion measurement system for risk perception of traffic participants based on the autonomous driving simulation platform Carla.

[0091] This embodiment includes interactive functions, data collection functions, data fusion functions and data analysis functions, and finally obtains a risk perception measurement model for road traffic users in simulation scenarios based on a physiological sensor system. Figure 3 shown.

[0092] (1) Interactive Function

[0093] Carla is an autonomous driving simulation software that provides open-source digital resources, such as buildings, vehicles, and other infrastructure, which can be used to construct the target road traffic scene and make it as close as possible to the actual road conditions expected to be studied during the scene design and construction stage. By adjusting the important influencing factors in the road traffic scene in Carla, such as environmental complexity, typical human and vehicle traffic, and environmental visibility, typical weather, morning and evening time, etc., different typical scenes can be designed and group control experiments can be conducted to control variables. At the same time, the operator can take the perspective of the driver and pedestrian respectively, and make behavioral decisions in the established scenes. This helps to fully understand the differences in cognition and decision-making of different road traffic users and the same road traffic user when switching identities.

[0094] The simulation platform for realizing interactive functions in the measurement system referred to in this embodiment is to build a typical road traffic scene with Carla as a tool, an external display with a resolution of 1080P as a visual interactive medium, and a keyboard as a behavioral interactive medium. The most critical implementation of the typical road traffic scene is divided into two parts, the service module end (Server) and the agent end (Client). The server end relies on the Unreal Engine (Ureal Engine) and is responsible for the presentation of the simulation, including presenting streets, buildings, traffic light change relationships, sensor calculation principles, etc., to present a complete simulation world. The agent end is responsible for inputting Python instructions in the application programming interface (Application Programming Interface, API) to update the world provided by the current server end, including weather changes, vehicle speed, number of pedestrians, etc. In order to provide experimental personnel interaction functions, this embodiment calls the Pygame toolkit in the API and designs motion control components for pedestrians and vehicles. Before the simulation scene finally presented on the screen, the experimental subject first obtains the voice task instruction by listening, further observes the scene by vision, determines the operation to be performed by itself through autonomous judgment and decision-making, and implements basic actions such as dragging the field of view, controlling the forward direction and speed, and emergency braking through the keyboard to complete the operation.

[0095] (2) Data collection function

[0096] This embodiment comprehensively considers EEG data and eye movement data, wherein EEG (Electroencephalogram, EEG) is a method of reflecting brain neural activity by measuring physiological electrical signals on the surface of the scalp, and EEG can be used for safety behavior research of drivers, pilots, etc.; eye tracking technology is used to track and measure eye position and movement information, which can be used to analyze individual visual characteristics such as gaze and eye saccades. Non-destructive eye tracking technology has been used in a large number of vision-related psychological and behavioral science studies; this embodiment proposes a multimodal collaborative solution of EEG and eye movement, and combines the advantages of the two technologies to provide a more comprehensive cognitive process analysis, and applies the measurement solution of brain-eye fusion to a virtual simulation platform experiment simulating road traffic scenes. It can induce various behavioral decisions in different road traffic scenes without causing any harm to the experimental subjects, and calculate various physiological indicators such as attention and vigilance with the help of multimodal data of brain-eye fusion to help understand their risk perception and behavioral mechanisms.

[0097] The data collection function in the measurement system referred to in this embodiment includes a questionnaire collection module, an EEG collection module, an eye movement collection module, an expression collection module, and a behavior collection module. The above modules can be arranged on different workstations, and it is necessary to use the same timing server to synchronize the system time in advance to ensure that each module can share a timestamp. Figure 4A feasible data collection scheme is given, in which the functions and implementation methods of each module are as follows:

[0098] 1) Questionnaire collection module: Before the experiment, basic information such as the traffic experience, personality, and risk tendency of the experimental subjects are collected; during the experiment, information such as cognitive load and immediate safety self-assessment of the experimental subjects are collected; after the experiment, information such as safety retrospective self-assessment of the experimental subjects are collected.

[0099] 2) EEG acquisition module: Use Emotiv Flex Saline 32-channel portable EEG with a sampling frequency of 120 Hz. The device is connected to the software EmotivPro via Bluetooth to transmit the sampled data in real time.

[0100] 3) Eye movement acquisition module: The portable screen-based eye tracker can receive and be compatible with various screen stimuli, and simultaneously record multiple data such as the subject's eye position, pupil size, eyelid closure, blinking behavior, gaze behavior, etc., in order to understand their visual way of perceiving stimuli and their decision-making response; using the Tobii pro fusion screen-based eye tracker with a sampling rate of 120Hz, through Tobii pro lab: with "visual tracking type" as the label, the corneal infrared reflection information of the subject when observing the screen is recorded throughout the process; with "keyboard event type" as the label, the start time and key type of all keyboard operations of the subject before, during and after the scene observation task are recorded; in the form of screen recording, all screen information before, during and after the experimental scene is run is recorded.

[0101] 4) Expression acquisition module: Use a camera with a resolution of 1080P and a frame rate of 30Hz, directly above the monitor, facing the face of the experimental subject, to record the facial expressions of the experimental subject in real time during the experiment.

[0102] 5) Behavior recording module: In order to ensure the integrity of the behavior data and improve the data precision, in addition to the behavior start time recorded by the above-mentioned TobiiPro Lab as "keyboard event type", this embodiment uses the Python toolkit logging (Logging) to record the keyboard key operations of the experimenter in the simulation scene presentation at a sampling rate of 30Hz, and stores it locally in txt file format.

[0103] exist Figure 4 In the data collection process, different signal receiving ends are aligned using the TUNA network time protocol. The specific operation process of this data collection solution is as follows:

[0104] Step 1. Align the network time protocol of each workstation.

[0105] Step 2. Eye tracking calibration.

[0106] A white dot was set on a black background, and the experimenter was instructed to visually track the white dot until it disappeared. The white dot appeared 9 times in total, appearing in the four corners and the center of the display screen in turn. The positions where all the dots appeared formed a centrally symmetrical figure.

[0107] Step 3. Deploy EEG equipment, detect signal quality, and collect EEG baselines.

[0108] Step 4. Start the eye tracker platform software project.

[0109] Install the Tobii Pro Lab program in advance and run it. After entering the license, create a new project file and design the recording components required for this experiment in the timeline. The recording components used in this embodiment are: vision correction component and screen recording component.

[0110] Step 5. Start facial expression recording.

[0111] Step 6. Start the carla server.

[0112] Configure Python 3.7 system environment on Windows computer in advance, download carla's unrealengine support files in advance, support files common appendix. Click to run CarlaUE4.exe file to start the basic simulation world.

[0113] Step 7. Start the API script.

[0114] In Pycharm, run the API script for the corresponding scenario.

[0115] Step 9. Start task interaction.

[0116] Step 10. The interaction ends.

[0117] The experimenter ends the interaction by pressing the backspace key, and the main experimenter shuts down the Carla server.

[0118] Step 11. The script closes.

[0119] The main tester closes the API script.

[0120] Step 12. Turn off data collection.

[0121] (3) Data fusion function

[0122] The realization of the data fusion function in the measurement system referred to in this embodiment firstly involves extracting the start and end time of the script in each single scene interaction task through the time instruction in the API, and extracting the EEG, eye movement, and expression data segments corresponding to the time period within the unified start and end timestamp range of the timing server. To ensure sufficient data analysis accuracy, the timestamps of the sampling points of multiple software terminals in this embodiment are accurate to milliseconds.

[0123] (4) Data analysis function

[0124] The establishment of the final risk perception measurement model in the measurement system referred to in this embodiment uses the risk perception questionnaire filled out by the experimental subjects during the experiment as the classification label, and uses EEG, eye movement, and emotion indicators as features. The specific feature extraction steps include:

[0125] (1) EEG indices: Bandpass filtering, re-referencing, EEG artifact removal based on independent component analysis (ICA), time domain / frequency domain conversion based on wavelet packet transform (WPT), signal segmentation and labeling, calculation of frequency domain energy during each task period, and finally calculation of EEG indices based on the frequency domain energy during the task period, including attention, vigilance, mental workload, mental fatigue, etc.

[0126] (2) Eye movement indicators: Data import and synchronization, calibration, filtering, artifact detection and removal are performed in sequence. After data segmentation and labeling, eye movement characteristics are extracted during each task period, including fixation duration, fixation point coordinates, saccade speed, saccade amplitude, blink frequency, etc. At the same time, the Area of ​​Interest (AOI) is defined, and the fixation time, number of times, and sequence on these areas are analyzed to understand the subject's attention to specific areas.

[0127] (3) Emotional indicators: Frame extraction is performed in sequence, and faces are detected and aligned using computer vision algorithms to standardize facial position and posture, and background and illumination noise is removed. The position, distance, and angle changes of facial key points (such as eye corners, eyebrows, and mouth corners, etc.) are extracted frame by frame, and facial muscle movements are analyzed. Local Binary Pattern (LBP) is used to monitor skin texture and subtle expression changes. According to the Facial Action Coding System (FACS), action units related to specific emotions are extracted and analyzed, such as eyebrow raising, eyelid contraction, and mouth corner raising. Finally, the above features are used to calculate the activation degree, duration, and frequency of basic emotional valence during the task period.

[0128] In order to implement the above embodiments, the present application also proposes a brain-eye fusion measurement method for risk perception of traffic participants based on a simulation platform.

[0129] Figure 5 A flowchart of a method for measuring the risk perception of traffic participants by brain-eye fusion based on a simulation platform is provided in an embodiment of the present application.

[0130] like Figure 5 As shown, the brain-eye fusion measurement method of traffic participant risk perception based on the simulation platform includes the following steps:

[0131] Step 501, obtaining risk perception questionnaire data filled out by traffic participants;

[0132] Step 502, collecting EEG data, eye movement data, facial expression data and behavior data of traffic participants when they interact with the simulation platform;

[0133] Step 503, calculating physiological indicators based on the collected data, wherein the physiological indicators include EEG indicators, eye movement indicators and emotion indicators;

[0134] Step 504, calculate the risk perception benchmark value based on the risk perception questionnaire data, use the risk perception benchmark value as a classification label of risk perception, use the physiological index as a feature, and construct a classification measurement model of risk perception.

[0135] Optionally, in one embodiment of the present application, it further includes:

[0136] Based on environmental variables, prior knowledge variables, unsafe behavior variables and perception degree variables, a structural equation model of risk perception-decision-making is established, in which the environmental variables are determined by traffic scene information, the prior knowledge variables are determined by the risk perception questionnaire data filled out by traffic participants, the unsafe behavior variables are determined by the behavior data of traffic participants, and the perception degree variables indicate the degree to which users perceive the surrounding environment information, and the perception degree variables are determined by physiological indicators.

[0137] Optionally, in one embodiment of the present application, the EEG data, eye movement data, expression data and behavior data of the traffic participants when interacting with the autonomous driving simulation platform are collected, including:

[0138] Use a portable EEG device to collect EEG data;

[0139] Use a screen-based eye tracker to collect eye movement data;

[0140] Use the camera to collect expression data;

[0141] The keyboard key operation data of the traffic participant is obtained, and the behavior data of the traffic participant is determined based on the keyboard key operation data.

[0142] Optionally, in one embodiment of the present application, the EEG index includes attention, vigilance, mental workload and mental fatigue level calculated based on time-frequency analysis and sub-band energy;

[0143] Eye movement metrics include fixation duration, fixation point coordinates, saccade velocity, saccade amplitude, and blink frequency extracted based on eye tracking;

[0144] Generate sentiment indicators, including:

[0145] Use computer vision algorithms to detect facial data and extract the position, distance, and angle changes of facial key points frame by frame;

[0146] Analyze facial muscle movements and detect skin texture and micro-expression changes using local binary patterns;

[0147] Extract and analyze action units associated with specific emotions using the Facial Action Coding System;

[0148] The activation degree, duration and frequency of basic emotional valence are calculated based on the position, distance and angle changes of facial key points, skin texture and micro-expression changes and action units.

[0149] It should be noted that the aforementioned explanation of the embodiment of the brain-eye fusion measurement system for risk perception of traffic participants based on a simulation platform is also applicable to the brain-eye fusion measurement method for risk perception of traffic participants based on a simulation platform of this embodiment, and will not be repeated here.

[0150] In order to implement the above embodiments, the present invention further proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method described in the above embodiments is implemented.

[0151] In order to implement the above embodiments, the present invention further proposes a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method of the above embodiments is implemented.

[0152] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0153] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0154] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0155] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0156] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0157] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0159] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A brain-eye fusion measurement system for traffic participant risk perception based on a simulation platform, characterized in that: include: A simulation platform, wherein the simulation platform is based on a simulated road traffic scene, uses an external display as a visual interaction medium, and uses a keyboard as a behavioral interaction medium to perform traffic scene simulation interaction with traffic participants; Questionnaire data collection module, used to obtain risk perception questionnaire data filled out by traffic participants; A physiological data collection module, used to collect EEG data, eye movement data, expression data and behavior data of traffic participants when they interact with the simulation platform; A physiological index calculation module, used to calculate physiological indexes based on the collected data, wherein the physiological indexes include EEG indexes, eye movement indexes and emotion indexes; The risk perception measurement module is used to calculate the risk perception benchmark value based on the risk perception questionnaire data, use the risk perception benchmark value as a classification label for risk perception, and use the physiological indicators as features to construct a classification measurement model for risk perception.

2. The system according to claim 1, characterized in that The risk perception measurement module is also used to establish a structural equation model of risk perception-decision-making based on environmental variables, prior knowledge variables, unsafe behavior variables and perception degree variables, wherein the environmental variables are determined by traffic scene information, the prior knowledge variables are determined by risk perception questionnaire data filled out by traffic participants, the unsafe behavior variables are determined by the behavior data of traffic participants, and the perception degree variable represents the degree to which the user perceives the surrounding environment information, and the perception degree variable is determined by the physiological indicators.

3. The system according to claim 1, characterized in that The physiological data acquisition module is specifically used for: Use a portable EEG device to collect EEG data; Use a screen-based eye tracker to collect eye movement data; Use the camera to collect expression data; The keyboard key operation data of the traffic participant is obtained, and the behavior data of the traffic participant is determined based on the keyboard key operation data.

4. The system according to claim 1, characterized in that The EEG indicators include attention, vigilance, mental workload and mental fatigue levels calculated based on time-frequency analysis and sub-band energy; The eye movement indicators include gaze duration, gaze point coordinates, saccade speed, saccade amplitude and blink frequency extracted based on eye movement tracking; Generating the sentiment indicator comprises: Use computer vision algorithms to detect facial data and extract the position, distance, and angle changes of facial key points frame by frame; Analyze facial muscle movements and detect skin texture and micro-expression changes using local binary patterns; Extract and analyze action units associated with specific emotions using the Facial Action Coding System; The basic emotional valence activation degree, duration and frequency are calculated based on the position, distance and angle changes of the facial key points, the skin texture and micro-expression changes and the action unit.

5. A brain-eye fusion measurement method for risk perception of traffic participants based on a simulation platform, characterized in that: include: Obtain risk perception questionnaire data filled out by traffic participants; Collecting EEG data, eye movement data, facial expression data and behavioral data of traffic participants when they interact with the simulation platform as claimed in claim 1; Calculating physiological indicators based on the collected data, wherein the physiological indicators include electroencephalogram indicators, eye movement indicators and emotion indicators; A risk perception benchmark value is calculated based on the risk perception questionnaire data, the risk perception benchmark value is used as a classification label of risk perception, and the physiological index is used as a feature to construct a classification measurement model of risk perception.

6. The method according to claim 5, characterized in that Also includes: Based on environmental variables, prior knowledge variables, unsafe behavior variables and perception degree variables, a structural equation model of risk perception-decision-making is established, wherein the environmental variables are determined by traffic scene information, the prior knowledge variables are determined by risk perception questionnaire data filled out by traffic participants, the unsafe behavior variables are determined by the behavior data of traffic participants, and the perception degree variable represents the degree to which the user perceives the surrounding environment information, and the perception degree variable is determined by the physiological indicators.

7. The method according to claim 5, characterized in that Collect EEG data, eye movement data, facial expression data, and behavioral data of traffic participants when they interact with the autonomous driving simulation platform, including: Use a portable EEG device to collect EEG data; Use a screen-based eye tracker to collect eye movement data; Use the camera to collect expression data; The keyboard key operation data of the traffic participant is obtained, and the behavior data of the traffic participant is determined based on the keyboard key operation data.

8. The method according to claim 5, characterized in that The EEG indicators include attention, vigilance, mental workload and mental fatigue levels calculated based on time-frequency analysis and sub-band energy; The eye movement indicators include gaze duration, gaze point coordinates, saccade speed, saccade amplitude and blink frequency extracted based on eye movement tracking; Generating the sentiment indicator comprises: Use computer vision algorithms to detect facial data and extract the position, distance, and angle changes of facial key points frame by frame; Analyze facial muscle movements and detect skin texture and micro-expression changes using local binary patterns; Extract and analyze action units associated with specific emotions using the Facial Action Coding System; The basic emotional valence activation degree, duration and frequency are calculated based on the position, distance and angle changes of the facial key points, the skin texture and micro-expression changes and the action unit.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 4 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 5 to 8 is implemented.

Images