Driver attention detection method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to a driver attention detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining at least one piece of eye movement data of the pupil position, the gazing direction and the eyelid opening and closing degree of the eyes of a driver, and carrying out the eye movement tracking of the at least one piece of eye movement data; environment elements of the environment where the vehicle is located are detected, and a region of interest of a driver is determined according to the environment elements; and calculating the deflection angle of the fixation point of the driver moving from the view center to the region of interest, evaluating the driving behavior of the driver according to the contact intersection point between the fixation line and the target observation element of the driver, and carrying out attention level quantitative evaluation on the driving behavior of the driver. Therefore, the problems that an attention testing method in the related technology cannot represent the relation with the attention, is high in subjectivity, lacks objectivity, is difficult to quantify the attention of the driver, reduces the driving safety, is low in intelligence and the like are solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for detecting driver attention. Background Art

[0002] Attention is one of the important abilities of human beings to perceive, understand and predict the surrounding environment. This concept was originally proposed in the military field and has been widely adopted in various disciplines due to its universal applicability. At present, it has also attracted more and more attention from academia and industry, especially in the field of autonomous driving. In autonomous driving, attention refers to the driver's ability to perceive, understand and predict road conditions, vehicles, pedestrians and other traffic participants, which forms the basis for the driver to assess the current traffic conditions and make appropriate decisions. Effective attention enables the driver to identify potential dangers in a timely manner, allowing the driver to make accurate driving decisions and improve overall driving safety. In autonomous driving tasks, attention is an important ability of the driver, especially in low-level autonomous driving systems involving collaborative driving between human drivers and autonomous driving systems. Determining whether the driver is able to complete the attention task or whether the autonomous driving system should set an attention task is crucial to achieving effective collaboration between the driver and the autonomous driving.

[0003] In the related technologies, the abstract concept of attention can be quantified into three types: the first is based on questionnaires, similar to the attention SAGAT (Situation Awareness Global Assessment Technique) and the attention SART (Situation Awareness Rating Technique). Most questionnaire survey methods require interrupting the experiment. These methods ask the subjects about the situation of their surroundings, or let the subjects recall certain scenes during the test afterwards. The second is based on behavioral observation, which evaluates the level of attention by observing the subjects' operating behaviors and decisions. It is also called the expert evaluation method (Subject Matter Experts, SME). The third is based on physiological indicators, including eye movement data, EEG, etc. These methods evaluate the impact of attention on humans through physiological data.

[0004] However, the attention testing methods in related technologies cannot characterize the relationship with attention, are highly subjective, lack objectivity, and are difficult to quantify the driver's attention, which reduces driving safety and has low intelligence, and is in urgent need of improvement. Summary of the invention

[0005] The present application provides a method, device, electronic device and storage medium for detecting driver's attention, so as to solve the problems that the attention testing method in the related art cannot characterize the relationship with attention, is highly subjective, lacks objectivity, is difficult to quantify the driver's attention, reduces driving safety, and has low intelligence.

[0006] The first aspect of the present application provides a method for detecting a driver's attention, comprising the following steps: obtaining at least one eye movement data of the driver's pupil position, gaze direction, and eyelid opening and closing degree, and performing eye tracking on the at least one eye movement data to obtain the driver's eye tracking result; based on the eye tracking result, detecting environmental elements of the vehicle's environment, and determining the driver's area of ​​interest based on the environmental elements; calculating the deflection angle of the driver's gaze point from the center of the field of view to the area of ​​interest, generating a gaze line based on the deflection angle, and evaluating the driver's driving behavior based on the contact intersection between the gaze line and the driver's target observation element, and performing a quantitative assessment of the driver's driving behavior to generate the driver's attention information.

[0007] Optionally, in one embodiment of the present application, determining the driver's area of ​​interest based on the environmental elements includes: acquiring at least one environmental element among a left rearview mirror, a right rearview mirror, a center rearview mirror, a dashboard, a road, surrounding vehicles and obstacles; determining the driver's attention concentration area based on the at least one environmental element, and determining the driver's area of ​​interest based on the attention concentration area.

[0008] Optionally, in one embodiment of the present application, the quantitative assessment of the driver's driving behavior attention level includes: based on a preset attention assessment algorithm, evaluating the driver's perceptual awareness level and information search ability according to an attention scoring mechanism, and obtaining the driver's attention perception scoring result; within a preset time, obtaining the driver's gaze duration on the target observation element to obtain the target observation element importance, and obtaining the driver's gaze frequency on the target observation element to obtain the driver's information understanding probability; obtaining the driver's attention understanding score according to the target observation element importance and the information understanding probability, so as to quantitatively assess the driver's driving behavior attention level according to the attention understanding score.

[0009] Optionally, in one embodiment of the present application, the calculation formula of the attention scoring mechanism is:

[0010]

[0011] Among them, n Hitis the number of elements observed by the driver, n total is the total number of elements present in the environment;

[0012] The calculation formula of the importance of the target observation element is:

[0013]

[0014] in, is the number of times the driver stays on an element, that is, the number of gaze points, is the average number of sampled gaze points for each gaze comprehension element;

[0015] The calculation formula of the information understanding probability is:

[0016]

[0017] Wherein, μ is the average expectation of the number of fixation points in a period of time, σ is the standard deviation of the number of fixation points in the period of time, and x is the count of the fixation points;

[0018] The calculation formula of the attention understanding score is:

[0019]

[0020] Among them, P i is the probability of understanding the information, μ importance Observe the element importance for the target.

[0021] Optionally, in one embodiment of the present application, it also includes: based on the changes in the direction and speed of the driver's eye movements, calculating the predictive ability of situational perception, and obtaining predictive information of the vehicle's environment based on the predictive ability; based on the predictive information, according to the attention scoring mechanism, the attention understanding score and the predictive ability of situational perception, obtaining the driver's total attention evaluation score.

[0022] Optionally, in one embodiment of the present application, the prediction capability formula of the context awareness is:

[0023]

[0024] Wherein, p(x) is the probability distribution of gaze direction in the field of view angle, p(y) is the probability distribution of gaze speed, and p(x, y) is the probability distribution of the gaze direction and the gaze speed at the same time.

[0025] A second aspect of the present application provides a device for detecting driver attention, comprising: an eye tracking module, used to obtain at least one eye movement data of the driver's pupil position, gaze direction and eyelid opening and closing degree, and perform eye tracking on the at least one eye movement data to obtain the driver's eye tracking result; a determination module, used to detect environmental elements of the vehicle's environment based on the eye tracking result, and determine the driver's area of ​​interest based on the environmental elements; a detection module, used to calculate the deflection angle of the driver's gaze point from the center of the field of view to the area of ​​interest, generate a gaze line based on the deflection angle, and evaluate the driver's driving behavior based on the contact intersection between the gaze line and the driver's target observation element, and perform a quantitative evaluation of the driver's driving behavior to generate the driver's attention information.

[0026] Optionally, in one embodiment of the present application, the determination module includes: an element acquisition unit, used to acquire at least one environmental element among a left rearview mirror, a right rearview mirror, a central rearview mirror, a dashboard, a road, surrounding vehicles and obstacles; a determination unit, used to determine the driver's attention focus area based on the at least one environmental element, and determine the driver's area of ​​interest based on the attention focus area.

[0027] Optionally, in one embodiment of the present application, the detection module also includes: an evaluation unit, which is used to evaluate the driver's perceptual awareness level and information search ability based on a preset attention evaluation algorithm and an attention scoring mechanism to obtain the driver's attention perception scoring result; a first acquisition unit, which is used to obtain the driver's gaze duration on the target observation element within a preset time to obtain the importance of the target observation element, and obtain the driver's gaze frequency on the target observation element to obtain the driver's information understanding probability; a second acquisition unit, which is used to obtain the driver's attention understanding score based on the target observation element importance and the information understanding probability, so as to quantitatively evaluate the attention level of the driver's driving behavior based on the attention understanding score.

[0028] Optionally, in one embodiment of the present application, the calculation formula of the attention scoring mechanism is:

[0029]

[0030] Among them, n Hit is the number of elements observed by the driver, n total is the total number of elements present in the environment;

[0031] The calculation formula of the importance of the target observation element is:

[0032]

[0033] in, is the number of times the driver stays on an element, that is, the number of gaze points, is the average number of sampled gaze points for each gaze comprehension element;

[0034] The calculation formula of the information understanding probability is:

[0035]

[0036] Wherein, μ is the average expectation of the number of fixation points in a period of time, σ is the standard deviation of the number of fixation points in the period of time, and x is the count of the fixation points;

[0037] The calculation formula of the attention understanding score is:

[0038]

[0039] Among them, P i is the probability of understanding the information, μ importance Observe the element importance for the target.

[0040] Optionally, in one embodiment of the present application, it also includes: a calculation module, which is used to calculate the prediction ability of situational perception based on the direction and speed changes of the driver's eye movements, and obtain prediction information of the vehicle's environment based on the prediction ability; an acquisition module, which is used to obtain the driver's total attention evaluation score based on the prediction information, according to the attention scoring mechanism, the attention understanding score and the prediction ability of situational perception.

[0041] Optionally, in one embodiment of the present application, the prediction capability formula of the context awareness is:

[0042]

[0043] Wherein, p(x) is the probability distribution of gaze direction in the field of view angle, p(y) is the probability distribution of gaze speed, and p(x, y) is the probability distribution of the gaze direction and the gaze speed at the same time.

[0044] A third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for detecting driver attention as described in the above embodiment.

[0045] A fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program that, when executed by a processor, implements the above-mentioned method for detecting driver attention.

[0046] The embodiment of the present application uses an infrared camera to track eye movements based on the spatial position information of the pupil position, gaze direction, and eyelid opening and closing, evaluates the driver's driving behavior based on the contact intersection between the gaze line and the driver's target observation element, and quantitatively evaluates the driver's driving behavior. This utilizes virtual reality technology and eye tracking technology to provide a non-interference method for quantifying attention, which is used to address the current gap in quantitative autonomous driving attention testing technology, and has a positive effect on improving autonomous driving testing, autonomous driving technology, and driver monitoring technology, improving driving safety, and high intelligence. As a result, the problems of the attention test method in the related technology that cannot characterize the relationship with attention, is highly subjective, lacks objectivity, is difficult to quantify the driver's attention, reduces driving safety, and has low intelligence are solved.

[0047] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0049] Figure 1 A flowchart of a method for detecting driver attention provided according to an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a driver's first-person perspective and sight tracking in a VR (Virtual Reality) virtual world according to a method for detecting driver attention in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a test scenario of a method for detecting driver attention according to an embodiment of the present application;

[0052] Figure 4 A schematic diagram of a built-in infrared eye tracking device for a method for detecting driver attention according to an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the structure of a driver attention detection device provided according to an embodiment of the present application;

[0054] Figure 6It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0056] The following describes the driver attention detection method, device, electronic device and storage medium of the embodiment of the present application with reference to the accompanying drawings. Aiming at the problem that the attention test method in the related art mentioned in the above background technology cannot characterize the relationship with attention, is highly subjective, lacks objectivity, is difficult to quantify the driver's attention, reduces driving safety, and has low intelligence, the present application provides a driver attention detection method, in which an infrared camera can be used to track the eye movement according to the spatial position information of the pupil position, gaze direction and eyelid opening and closing degree of the eye, and the driver's driving behavior is evaluated according to the contact intersection between the gaze line and the driver's target observation element, and the driver's driving behavior is quantitatively evaluated for the attention level, thereby utilizing virtual reality technology and line of sight tracking technology, providing a non-interference method for quantifying attention, which is used to solve the gap in the current quantitative automatic driving attention test technology, and has a positive effect on improving automatic driving tests, automatic driving technology and driver monitoring technology, improving driving safety, and having high intelligence. Thus, the problem that the attention test method in the related art cannot characterize the relationship with attention, is highly subjective, lacks objectivity, is difficult to quantify the driver's attention, reduces driving safety, and has low intelligence is solved.

[0057] Specifically, Figure 1 A flowchart of a method for detecting driver attention provided in an embodiment of the present application.

[0058] like Figure 1 As shown, the driver attention detection method includes the following steps:

[0059] In step S101, at least one eye movement data of the driver's pupil position, gaze direction and eyelid opening and closing degree is obtained, and eye movement tracking is performed on the at least one eye movement data to obtain the driver's eye movement tracking result.

[0060] It is understandable that the eye tracking in the embodiments of the present application can be performed with the aid of gaze tracking technology.

[0061] During the actual implementation process, the embodiments of the present application can obtain eye movement data such as the spatial position information of the driver's pupil position, gaze direction, and eyelid opening and closing degree, and perform eye movement tracking based on the eye movement data through an infrared camera to obtain the driver's eye movement tracking results, thereby providing support for subsequent evaluation of the driver's attention level and thereby improving driving safety.

[0062] In step S102, based on the eye tracking result, the environmental elements of the vehicle's environment are detected, and the driver's area of ​​interest is determined according to the environmental elements.

[0063] It is understandable that the embodiments of the present application can analyze the driver's visual features, such as the division of different areas and the movement of eyes between these areas, so as to ensure that the visual scan is given semantic meaning through the actual AOIs (Areas of Interests).

[0064] In the actual implementation process, the embodiment of the present application can obtain and detect environmental elements of the vehicle's environment based on eye tracking results, and establish a ROI (Region of Interest) based on the environmental elements, thereby determining the driver's area of ​​interest, in which all objects are marked.

[0065] The embodiments of the present application can determine the driver's area of ​​interest based on environmental elements, thereby improving driving safety, comprehensively evaluate current driving conditions based on environmental elements, and make more reasonable and safe driving decisions.

[0066] Optionally, in one embodiment of the present application, determining the driver's area of ​​interest based on environmental elements includes: acquiring at least one environmental element among a left rearview mirror, a right rearview mirror, a central rearview mirror, a dashboard, a road, surrounding vehicles and obstacles; determining the driver's attention concentration area based on at least one environmental element, and determining the driver's area of ​​interest based on the attention concentration area.

[0067] It can be understood that the left rearview mirror, right rearview mirror, and center rearview mirror in the embodiment of the present application provide the driver with visual information of the vehicle's surroundings; the instrument panel in the embodiment of the present application can provide information such as speed, fuel level, and engine temperature; the road in the embodiment of the present application includes information such as lane lines, traffic signs, and road conditions, which is the basis for driving decisions; the surrounding vehicles in the embodiment of the present application include vehicles in front, parallel vehicles, and vehicles seen from behind through the rearview mirror, etc., and the vehicles can affect the driver's driving decisions and actions; the obstacles in the embodiment of the present application include but are not limited to pedestrians, animals, and dropped objects, etc.

[0068] In the actual implementation process, Figure 2As shown, the left rearview mirror, right rearview mirror and central rearview mirror in the embodiment of the present application play a key role in providing important visual information of the surrounding area, and the instrument panel is used to display important indicators such as vehicle speed, autonomous driving status and attention requests accompanied by alarms, and roads, vehicles and obstacles are particularly important in the driving environment, which are clearly defined, wherein each vehicle is regarded as an independent element. Therefore, the embodiment of the present application can obtain at least one environmental element among the left rearview mirror, the right rearview mirror, the central rearview mirror, the instrument panel, the road, the surrounding vehicles and obstacles, determine the driver's attention focus area according to the at least one environmental element, and determine the driver's area of ​​interest according to the attention focus area.

[0069] The embodiment of the present application can determine the driver's attention concentration area based on at least one environmental element, and determine the driver's area of ​​interest based on the attention concentration area, so that the driver can allocate his attention resources more accurately, further improve driving safety, and improve driving efficiency.

[0070] In step S103, the deflection angle of the driver's gaze point from the center of the field of view to the area of ​​interest is calculated, a gaze line is generated according to the deflection angle, and the driver's driving behavior is evaluated according to the contact intersection between the gaze line and the driver's target observation element. The driver's driving behavior is quantitatively evaluated for the attention level, and the driver's attention information is generated.

[0071] It can be understood that the target observation element of the driver in the embodiment of the present application may be an element observed by the driver in an autonomous driving monitoring task in a virtual environment.

[0072] In the actual implementation process, the embodiment of the present application can calculate the deflection angle of the driver's gaze point from the center of the visual field to the area of ​​interest, generate a gaze line according to the deflection angle, detect the elements observed in the autonomous driving monitoring task according to the gaze line, and evaluate the driver's driving behavior according to the contact intersection between the gaze line and the driver's target observation element, wherein, Figure 2 As shown, the intersection indicates the contact between the gaze line and the circle in the virtual environment. The embodiment of the present application can understand the driver's visual attention allocation and scanning strategy in a specific driving scenario based on the distribution of spatial line of sight data, thereby facilitating the evaluation of driving behavior. By quantitatively evaluating the driver's attention level of the driving behavior, the driver's attention information is generated.

[0073] The embodiments of the present application use a virtual method to analyze gaze interactions more efficiently and specifically, thereby greatly enhancing the understanding of human situational perception and behavior.

[0074] Optionally, in one embodiment of the present application, a quantitative assessment of the driver's driving behavior is performed on the attention level, including: based on a preset attention assessment algorithm, evaluating the driver's perceptual awareness level and information search ability according to an attention scoring mechanism, and obtaining the driver's attention perception scoring result; within a preset time, obtaining the driver's gaze duration on the target observation element to obtain the importance of the target observation element, and obtaining the driver's gaze frequency on the target observation element to obtain the driver's information understanding probability; obtaining the driver's attention understanding score based on the target observation element importance and the information understanding probability, so as to quantitatively assess the driver's driving behavior based on the attention understanding score.

[0075] It is understandable that attention perception can also be regarded as a comprehensive understanding of the surrounding environment, in which the identification of elements serves as the initial step of the perception process. In this process, the interaction between the gaze line and the virtual environment provides information about the driver's information search process and attention allocation. In the virtual environment, the more elements the gaze line contacts, the higher the possibility of cognition of the entire environment. Therefore, this helps to have a deeper understanding of the perception layer in situational cognition. The preset duration in the embodiment of the present application can be a specified time range.

[0076] In the actual implementation process, the embodiment of the present application can introduce the concept of attention scoring based on the preset attention evaluation algorithm, evaluate the driver's perceptual awareness level and information search ability according to the attention scoring mechanism, and obtain the driver's attention perception scoring result. The embodiment of the present application can obtain the driver's gaze duration on the target observation element within a specified time range, obtain the importance of the target observation element, and obtain the driver's gaze frequency on the target observation element to obtain the driver's information understanding probability, and obtain the driver's attention understanding score based on the target observation element importance and information understanding probability, so as to quantitatively evaluate the driver's driving behavior based on the attention understanding score.

[0077] It should be noted that the preset attention evaluation algorithm and the preset time can be set by those skilled in the art according to actual conditions and are not specifically limited here.

[0078] Optionally, in one embodiment of the present application, the calculation formula of the attention scoring mechanism is:

[0079]

[0080] Among them, n Hit is the number of elements observed by the driver, n total is the total number of elements present in the environment;

[0081] The calculation formula for the importance of the target observation element is:

[0082]

[0083] in, is the number of times the driver stays on an element, that is, the number of gaze points, is the average number of sampled gaze points for each gaze comprehension element;

[0084] The formula for calculating the probability of information understanding is:

[0085]

[0086] Where μ is the average expectation of the number of fixations in a period of time, σ is the standard deviation of the number of fixations in a period of time, and x is the count of fixations;

[0087] The calculation formula for the attention comprehension score is:

[0088]

[0089] Among them, P i is the probability of understanding the information, μ importance Observe the element importance for the target.

[0090] Specifically, the present embodiment introduces the concept of attention score, which is represented by SA L1 ,in:

[0091]

[0092] This score is simply defined as the proportion of elements that collide with the fixation line within a specified time frame. By analyzing the relative proportion of collisions with fixation objects, the driver's perceptual awareness level and information search ability can be evaluated. Environmental elements related to the driving task provide drivers with a lot of information and are considered to be critical for understanding the importance of virtual elements and the entire environment. In addition, the attention perception score can be applied to a variety of different driving scenarios to identify patterns and trends in human perception.

[0093] Furthermore, the attention perception in the embodiments of the present application includes not only the perception of elements in the environment, but also the understanding of the information conveyed by these elements and their meaning in the context of the environment. This understanding includes various attributes, such as size, distance, speed, and other relevant characteristics. In order to evaluate the level of understanding in situational awareness, two aspects of the understanding layer are abstracted: 1) the probability of understanding the information associated with each element, and 2) the importance of the information conveyed by each element. Therefore, the attention understanding score is defined as the sum of the probability of understanding the element and the importance of the element, expressed as in:

[0094]

[0095] Furthermore, in order to quantify the probability of understanding, the embodiment of the present application can use the normal distribution probability density function for modeling, expressed as μ importance ,in:

[0096]

[0097] Therefore, if the fixation duration deviates further from the average, the probability of understanding the current situation decreases. This approach is adopted because abnormally long or short fixation durations may indicate difficulties in understanding the information of the element. For understanding the current environment, the importance of each element can be characterized by the fixation frequency of the element. Therefore, the embodiment of the present application has a higher probability of understanding the event P. i , expressed by the probability density function expression of the normal distribution, where:

[0098]

[0099] Too long or too short a fixation duration indicates difficulty in understanding an event or unconscious distraction, and the probability of understanding an event in the current environment approaches 0, which means that the farther away from the mean, the smaller the probability of understanding an event.

[0100] Furthermore, within a certain period of time, the embodiment of the present application can obtain the total number of gazes on an element. Similarly, the number of gazes on each element is divided by the average number of gazes on all elements, which is called μ importance , i.e. importance. This concept indicates a greater emphasis on specific elements than other elements in the environment. With a comprehensive understanding of the environment, the driver can make more informed decisions in decision making. This situational understanding score ensures that the most critical and relevant information is prominently considered, further enhancing situational awareness and facilitating an effective decision-making process.

[0101] Furthermore, the importance of understanding the current environment can be represented by the fixation duration. The number of fixations on an element can be recorded as That is, the number of gaze points. In unit time, the embodiment of the present application can obtain the average number of sampled gaze points of each gaze element, which is recorded as Similarly, in the embodiment of the present application, the number of sampled gaze points for understanding each element can be divided by the average number of sampled gaze points to obtain the importance of the current element relative to all observed elements, which is denoted as: μ importance .

[0102] The embodiment of the present application improves the accuracy of calculation based on the formula, thereby further effectively detecting the driver's attention, which has a positive effect on improving autonomous driving tests, improving autonomous driving technology and driver monitoring technology.

[0103] Optionally, in one embodiment of the present application, it also includes: calculating the predictive ability of situational perception based on the changes in the direction and speed of the driver's eye movements, and obtaining predictive information of the vehicle's environment based on the predictive ability; based on the predictive information, obtaining the driver's total attention evaluation score according to the attention scoring mechanism, the attention understanding score and the predictive ability of situational perception.

[0104] It is understandable that in situational perception, the prediction of the environment refers to a higher-level representation of situational perception. Drivers can make decisions about future states by perceiving and understanding the environment. This predictive ability is reflected in human eye movement patterns, including visual fixation and scanning patterns, and the calculation of statistical gaze entropy and gaze transfer entropy is completed by using the statistical gaze discrete distribution coordinates in space to generate probability distributions. These methods analyze the randomness and complexity of eye movement patterns and provide practical guidance for prediction.

[0105] In the actual implementation process, at the prediction layer, the eye movement of each gaze ray can be abstracted into changes in direction and speed, such as Figure 2 As shown, point P represents the gaze ray collision in the virtual environment, t represents the time series, and the direction and speed changes between the two gaze ray collision points can be calculated by the sine function and the Euclidean distance. In order to calculate the predictive ability of situational awareness, the embodiment of the present application utilizes the mutual information of the joint distribution of the direction p(x) and speed p(y) sequences. Using joint information to evaluate predictive ability is a widely used method, and its robustness and accuracy have been widely verified, ensuring that the embodiment of the present application can calculate the predictive ability of situational awareness based on the direction and speed changes of the driver's eye movements, and obtain predictive information of the vehicle's environment based on the predictive ability, allowing the prediction of future dynamic changes based on historical gaze vectors. Based on the predicted information, according to the attention scoring mechanism, the attention understanding score and the predictive ability of situational awareness, the driver's total attention evaluation score is obtained.

[0106] The embodiments of the present application utilize virtual reality technology and gaze tracking technology to provide a non-interference method for quantifying attention, which is used to address the current gaps in quantitative autonomous driving attention testing technology and has a positive effect on improving autonomous driving testing, autonomous driving technology, and driver monitoring technology.

[0107] In one embodiment of the present application, the prediction capability formula of context awareness is:

[0108]

[0109] Among them, p(x) is the probability distribution of gaze direction in the field of view (FoV), p(y) is the probability distribution of gaze speed, and p(x,y) is the probability distribution of gaze direction and gaze speed at the same moment.

[0110] According to the formula, the embodiment of the present application can further improve the accuracy of the calculation and provide support for effectively detecting the driver's attention ability.

[0111] In order to verify the effectiveness of this application, three common autonomous driving attention scenarios are constructed, such as Figure 3 As shown, to calculate whether the proposed attention index can evaluate the driver's attention state.

[0112] The attention index and common driving stability and safety evaluation indicators such as lane departure velocity, average acceleration velocity, and collision rate were tested and verified through the Pearson correlation coefficient and p-value.

[0113] Table 1 is a table of test results. Through correlation analysis with common attention evaluation indicators, it can be seen that the evaluation indicators constructed in this application are significant in terms of lane departure speed standard deviation, average acceleration standard deviation, whether a collision occurs, and other indicators.

[0114] (1) The lane departure speed standard deviation is negatively correlated with the constructed attention evaluation index. That is, as the attention index increases, the standard deviation of the lane departure speed becomes smaller, which proves that the driver is more stable when operating the vehicle to change lanes.

[0115] (2) The standard deviation of the average acceleration is negatively correlated with the constructed attention evaluation index. Although its correlation index does not reach a strong correlation of 0.6, it also shows a certain negative correlation, that is, as the attention index increases, the standard deviation of the average acceleration gradually decreases, proving that the driver can detect danger earlier and perform emergency braking operations when performing emergency braking, making the change in acceleration less obvious.

[0116] (3) The probability of collision is negatively correlated with the constructed attention evaluation index, that is, as the attention index increases, the probability of collision decreases, proving that the more focused the driver's attention is, the more likely he is to avoid a collision. As shown in Table 1.

[0117] Table 1

[0118]

[0119] Among them, * represents p < 0.05, and ** represents p < 0.005.

[0120] Specifically, it can be combined Figures 2 to 4 As shown, the working principle of the driver attention detection method in the embodiment of the present application is described in detail with a specific embodiment.

[0121] In the hardware of the present application embodiment, Figure 4 As shown, an HTC ProEye headset (or a head-mounted VR device with the same accuracy) equipped with a 0.2-degree accurate infrared eye tracker is required to ensure that the participants are provided with at least 87 degrees of horizontal and vertical field of view. The recommended computer configuration is Intel 12th generation (or above) CPU, 16GB of memory (or more) and GPU NVIDIA GeForce RTX 30 series or 40 series graphics card. The driving simulator recommends a driving simulator set including a steering wheel, accelerator pedal and brake pedal. These hardware devices should be able to ensure that the project runs at a stable 60 frames per second.

[0122] In the test scenario of the embodiment of the present application, it is recommended to use Unity to build the virtual environment. Figure 2 It shows the driver’s first-person perspective and line-of-sight tracking in the VR virtual world. Figure 3 An overhead view of the virtual environment test scene is shown. In this test, all vehicles maintained the same speed in each lane, and the participant's vehicle was in the middle lane. At the beginning of each trial, the autonomous driving function was activated by default. (The autonomous driving function can be implemented in a variety of ways: such as software-in-the-loop, hardware-in-the-loop, vehicle-in-the-loop, etc.). In order to ensure that long-term monitoring of autonomous driving does not reduce the driver's vigilance, this application sets a 2-minute autonomous driving time before the attention scene. After 2 minutes, the autonomous driving vehicle will encounter one of three situations, namely overtaking, emergency braking, and right insertion. According to previous research results, the average visual search time for preparing to change lanes is about 6.6 seconds. Therefore, TOR will be set to TTC = 7 seconds to cancel the autonomous driving function by stepping on the brakes or pressing the button, that is, the initial timing of attention. Due to the existence of an operational design domain for the low-level autonomous driving vehicle tested in this test, the situations that the autonomous driving system cannot handle when not handled will be described as follows:

[0123] Overtaking: On a highway with moderate traffic density, a slow vehicle is found occupying the road. The autonomous driving system will slow down and eventually keep a proper distance from the vehicle in front. However, the autonomous driving system cannot find the right time to overtake the slow vehicle.

[0124] Emergency Braking: On a highway with moderate traffic density, a rear-end collision occurred in the lane. The autonomous driving system would slow down and eventually maintain an appropriate distance from the vehicle in front. However, the autonomous driving system could not handle the situation at the scene of the accident.

[0125] Cutting in from the right: On a highway with moderate traffic density, a vehicle on the right cuts in for no apparent reason and slows down. The Autopilot system slows down and eventually maintains an appropriate distance from the cutting vehicle. However, the Autopilot system finds that the cutting vehicle is too close.

[0126] In the test process of the embodiment of the present application, before the experiment, the purpose of the experiment was briefly introduced to the participants, and they signed a consent form. After filling in the form, the subjects were guided to the driving simulator; then, they were asked to sit in the driver's seat, adjust the seat and get a VR helmet, and complete the eye tracker calibration procedure under guidance; then, they were asked to drive the vehicle in a virtual environment for 5-10 minutes. During the experience, the participants were asked whether they felt uncomfortable and whether they were familiar with the behavior of the vehicle. In addition, the present application will also introduce car functions to the driver, such as how to activate the autonomous driving function and how to turn off the autonomous driving function. After the tutorial, the participants were asked to take off their helmets and rest for 2-3 minutes. After another round of eye tracker calibration, all participants completed the specified 3 attention scenarios. Between each trial, the present application required the driver to close his eyes and rest for 1 minute, but they were not allowed to take off their helmets.

[0127] According to the driver attention detection method proposed in the embodiment of the present application, eye tracking can be performed based on the spatial position information of the pupil position, gaze direction and eyelid opening and closing through an infrared camera, and the driver's driving behavior can be evaluated based on the contact intersection between the gaze line and the driver's target observation element, and the driver's driving behavior can be quantitatively evaluated for the attention level, thereby utilizing virtual reality technology and line of sight tracking technology to provide a non-interference method for quantifying attention, which is used to solve the current gap in quantitative automatic driving attention testing technology, and has a positive effect on improving automatic driving testing, automatic driving technology and driver monitoring technology, and improves driving safety and intelligence. As a result, the problem that the attention test method in the related technology cannot characterize the relationship with attention, is highly subjective, lacks objectivity, is difficult to quantify the driver's attention, reduces driving safety, and has low intelligence is solved.

[0128] Next, a device for detecting driver attention according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0129] Figure 5 It is a structural schematic diagram of a device for detecting driver attention according to an embodiment of the present application.

[0130] like Figure 5 As shown, the driver attention detection device 10 includes: an eye tracking module 100 , a determination module 200 and a detection module 300 .

[0131] Specifically, the eye tracking module 100 is used to obtain at least one eye movement data of the driver's pupil position, gaze direction and eyelid opening and closing degree, and perform eye movement tracking on the at least one eye movement data to obtain the driver's eye movement tracking result.

[0132] The determination module 200 is used to detect environmental elements of the vehicle's environment based on the eye tracking results, and determine the driver's area of ​​interest according to the environmental elements.

[0133] The detection module 300 is used to calculate the deflection angle of the driver's gaze point from the center of the visual field to the area of ​​interest, generate a gaze line according to the deflection angle, evaluate the driver's driving behavior according to the contact intersection between the gaze line and the driver's target observation element, and quantify the attention level of the driver's driving behavior to generate the driver's attention information.

[0134] Optionally, in one embodiment of the present application, the determination module 200 includes: an element acquisition unit and a determination unit.

[0135] The element acquisition unit is used to acquire at least one environmental element from among a left rearview mirror, a right rearview mirror, a central rearview mirror, a dashboard, a road, surrounding vehicles and obstacles;

[0136] The determining unit is used to determine the driver's attention concentration area according to at least one environmental element, and determine the driver's area of ​​interest according to the attention concentration area.

[0137] Optionally, in one embodiment of the present application, the detection module 300 further includes: an evaluation unit, a first acquisition unit and a second acquisition unit.

[0138] Among them, the evaluation unit is used to evaluate the driver's perception awareness level and information search ability based on a preset attention evaluation algorithm and an attention scoring mechanism to obtain the driver's attention perception scoring result.

[0139] The first acquisition unit is used to obtain the driver's gaze duration on the target observation element within a preset time to obtain the importance of the target observation element, and to obtain the driver's gaze frequency on the target observation element to obtain the driver's information understanding probability.

[0140] The second acquisition unit is used to obtain the driver's attention understanding score according to the importance of the target observation element and the information understanding probability, so as to quantitatively evaluate the driver's driving behavior attention level according to the attention understanding score.

[0141] Optionally, in one embodiment of the present application, the calculation formula of the attention scoring mechanism is:

[0142]

[0143] Among them, n Hit is the number of elements observed by the driver, n total is the total number of elements present in the environment;

[0144] The calculation formula for the importance of the target observation element is:

[0145]

[0146] in, is the number of times the driver stays on an element, that is, the number of gaze points, is the average number of sampled gaze points for each gaze comprehension element;

[0147] The formula for calculating the probability of information understanding is:

[0148]

[0149] Where μ is the average expectation of the number of fixations in a period of time, σ is the standard deviation of the number of fixations in a period of time, and x is the count of fixations;

[0150] The calculation formula for the attention comprehension score is:

[0151]

[0152] Among them, P i is the probability of understanding the information, u importance Observe the element importance for the target.

[0153] Optionally, in one embodiment of the present application, the driver attention detection device 10 further includes: a calculation module and an acquisition module.

[0154] Among them, the calculation module is used to calculate the prediction ability of situational perception based on the direction and speed changes of the driver's eye movements, and obtain prediction information of the vehicle's environment based on the prediction ability.

[0155] The acquisition module is used to obtain the driver's total attention evaluation score based on the prediction information according to the attention scoring mechanism, the attention understanding score and the prediction ability of the situational perception.

[0156] Optionally, in one embodiment of the present application, the prediction capability formula of context awareness is:

[0157]

[0158] Among them, p(x) is the probability distribution of gaze direction in the field of view, p(y) is the probability distribution of gaze speed, and p(x,y) is the probability distribution of gaze direction and gaze speed at the same time.

[0159] It should be noted that the above explanation of the embodiment of the method for detecting driver's attention is also applicable to the device for detecting driver's attention of this embodiment, and will not be repeated here.

[0160] According to the driver attention detection device proposed in the embodiment of the present application, eye tracking can be performed based on the spatial position information of the pupil position, gaze direction and eyelid opening and closing through an infrared camera, and the driver's driving behavior can be evaluated based on the contact intersection between the gaze line and the driver's target observation element, and the driver's driving behavior can be quantitatively evaluated for the attention level, thereby utilizing virtual reality technology and line of sight tracking technology to provide a non-interference method for quantifying attention, which is used to solve the current gap in quantitative automatic driving attention testing technology, and has a positive effect on improving automatic driving testing, improving automatic driving technology and driver monitoring technology, improving driving safety, and having high intelligence. As a result, the problem that the attention test method in the related technology cannot characterize the relationship with attention, is highly subjective, lacks objectivity, is difficult to quantify the driver's attention, reduces driving safety, and has low intelligence is solved.

[0161] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0162] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0163] When the processor 602 executes the program, the driver attention detection method provided in the above embodiment is implemented.

[0164] Furthermore, the electronic device further comprises:

[0165] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0166] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0167] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0168] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0169] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0170] The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0171] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned method for detecting driver attention is implemented.

[0172] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0173] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0174] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0175] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0176] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0177] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0178] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0179] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for detecting driver attention, characterized in that: The following steps are involved: Acquire at least one eye movement data of the driver's pupil position, gaze direction, and eyelid opening and closing degree, and perform eye movement tracking on the at least one eye movement data to obtain an eye movement tracking result of the driver; Based on the eye tracking result, detecting environmental elements of the vehicle's environment, and determining the driver's area of ​​interest according to the environmental elements; The method comprises calculating a deflection angle of the driver's gaze point from the center of the visual field to the area of ​​interest, generating a gaze line according to the deflection angle, evaluating the driver's driving behavior according to a contact intersection point between the gaze line and the driver's target observation element, and performing a quantitative evaluation of the driver's driving behavior and generating the driver's attention information.

2. The method for detecting driver attention according to claim 1, characterized in that: The determining the driver's area of ​​interest according to the environmental elements comprises: Acquire at least one environmental element among a left rearview mirror, a right rearview mirror, a central rearview mirror, a dashboard, a road, surrounding vehicles, and obstacles; An attention concentration area of ​​the driver is determined according to the at least one environmental element, and an area of ​​interest of the driver is determined according to the attention concentration area.

3. The method for detecting driver attention according to claim 1, characterized in that: The quantitative assessment of the driver's driving behavior attention level includes: Based on a preset attention evaluation algorithm, the driver's perception awareness level and information search ability are evaluated according to an attention scoring mechanism to obtain an attention perception scoring result of the driver; Within a preset time, the driver's gaze duration on the target observation element is obtained to obtain the importance of the target observation element, and the driver's gaze frequency on the target observation element is obtained to obtain the driver's information understanding probability; The driver's attention understanding score is obtained according to the target observation element importance and the information understanding probability, so as to quantitatively evaluate the attention level of the driver's driving behavior according to the attention understanding score.

4. The method for detecting driver attention according to claim 3, characterized in that: The calculation formula of the attention scoring mechanism is: Among them, n Hit is the number of elements observed by the driver, n total is the total number of elements present in the environment; The calculation formula of the importance of the target observation element is: in, is the number of times the driver stays on an element, that is, the number of gaze points, is the average number of sampled gaze points for each gaze comprehension element; The calculation formula of the information understanding probability is: Wherein, μ is the average expectation of the number of fixation points in a period of time, σ is the standard deviation of the number of fixation points in the period of time, and x is the count of the fixation points; The calculation formula of the attention understanding score is: Among them, P i is the probability of understanding the information, μ importance Observe the element importance for the target.

5. The method for detecting driver attention according to claim 3, characterized in that: Also includes: Calculating the prediction capability of situational awareness based on the changes in the direction and speed of the driver's eye movements, and obtaining prediction information of the environment in which the vehicle is located according to the prediction capability; Based on the prediction information, the total attention evaluation score of the driver is obtained according to the attention scoring mechanism, the attention understanding score and the prediction ability of the situational awareness.

6. The method for detecting driver attention according to claim 5, characterized in that: The predictive power formula of the situational awareness is: Wherein, p(x) is the probability distribution of gaze direction in the field of view angle, p(y) is the probability distribution of gaze speed, and p(x, y) is the probability distribution of the gaze direction and the gaze speed at the same time.

7. A driver attention detection device, characterized in that: include: An eye tracking module, used to obtain at least one eye movement data of the driver's pupil position, gaze direction and eyelid opening and closing degree, and perform eye movement tracking on the at least one eye movement data to obtain an eye movement tracking result of the driver; A determination module, configured to detect environmental elements of the vehicle's environment based on the eye tracking result, and determine the driver's area of ​​interest according to the environmental elements; A detection module is used to calculate the deflection angle of the driver's gaze point from the center of the visual field to the area of ​​interest, generate a gaze line according to the deflection angle, evaluate the driver's driving behavior according to the contact intersection between the gaze line and the driver's target observation element, and quantitatively evaluate the driver's driving behavior. Attention level, generate attention information of the driver.

8. The driver attention detection device according to claim 7, characterized in that: The determination module comprises: an element acquisition unit, used to acquire at least one environmental element among a left rearview mirror, a right rearview mirror, a central rearview mirror, a dashboard, a road, surrounding vehicles and obstacles; A determination unit is used to determine the driver's attention concentration area according to the at least one environmental element, and determine the driver's area of ​​interest according to the attention concentration area.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for detecting driver attention as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the driver attention detection method as described in any one of claims 1-6.

Images