A safe driving recommendation method in urban environment based on driver's eye movement

Abstract

The present invention discloses a method for recommending safe driving in an urban environment based on the driver's eye movement scanning, comprising the following steps: collecting driving videos of subjects, obtaining the gaze points of the subjects and a model, and obtaining key features using a classical attention model; constructing a neural network, selecting the most appropriate gaze point; and determining a final recommended safe driving plan by optimizing the gaze point. In a crowded road environment, the present invention can improve the accuracy of scanning by comparing the differences in the scanning trajectories between the subjects and the model, meet safety requirements, recommend safe driving plans, achieve safe driving, and provide a reference for future fully autonomous driving.

Description

Technical Field

[0001] The present invention relates to driver attention analysis and modeling technology, and in particular to a safe driving recommendation method in an urban environment based on driver eye movement scanning. Background Art

[0002] In recent decades, computational modeling of human visual attention has received extensive attention. In contemporary advanced industrial applications, it has been shown to be able to predict human visual attention very well. However, there has been controversy over whether the eye saccades predicted by computational visual attention models (Saccades Predicted by Computational Visual Attention Model) and human visual attention-based eye saccades are more reliable and whether they are actually helpful for actual driving.

[0003] Considering that biological inspiration for driving-related attention models can be obtained from skilled drivers in complex driving conditions, where drivers' attention is constantly directed to various salient and informative visual stimuli through alternating eye fixations for safe driving, this paper proposes a saccade recommendation strategy to improve driving safety in crowded road environments, especially when drivers' vision is often affected by visual crowding. Summary of the invention

[0004] In view of this, the present invention provides a safe driving recommendation method in an urban environment based on the driver's eye movement scanning to solve the above technical problems.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A safe driving recommendation method in an urban environment based on driver eye movement scanning comprises the following steps:

[0007] Step 1: Collection and preprocessing of original driving videos: collecting original driving videos of virtual and real urban driving scenes, and preprocessing the original driving videos;

[0008] Step 2: Obtain the eye movement trajectories and fixation points of the subjects and the model, recruit subjects with driving experience to participate in the eye movement experiment, obtain the common eye movement trajectories of the subjects, and obtain the common eye movement trajectories of the subjects through the eye movement saccade device; select an attention model to predict the fixation points and corresponding saccade trajectories of the preprocessed driving video;

[0009] Step 3: Preliminary selection of the scanning trajectory scheme. By comparing the differences in the fixation points and scanning of the subjects and the model, the optimal scanning trajectory scheme is preliminarily selected;

[0010] Step 4: Extract key features related to eye movements in the preprocessed raw driving video;

[0011] Step 5: Build a neural network model and select the most reasonable fixation point. Build a neural network model and select the most reasonable fixation point by referring to the overall safe driving plan to meet the requirements of safe driving.

[0012] Step 6: Optimize the fixation point in step 5 and determine the final recommended safe driving plan.

[0013] Furthermore, the acquisition of original driving videos for virtual and real urban driving scenes includes: half of the original driving videos are shot in real scenes, and the other half of the original driving videos are shot in virtual scenes, the videos in real scenes in the original driving videos are shot using a digital camera installed on the car windshield, and the videos in virtual scenes in the original driving videos are shot using a high-definition camera installed on a tripod.

[0014] Furthermore, the digital camera installed on the car windshield records driving videos of 10 different driving environments in real scenes, and extracts driving video clips corresponding to 6 different driving environments therefrom, and the high-definition camera installed on the tripod records driving videos of 10 different driving environments in virtual scenes, and extracts driving video clips corresponding to 6 different driving environments therefrom.

[0015] Furthermore, the preprocessing of the original driving video includes performing color thresholding processing and Gaussian filtering processing on the original driving video.

[0016] Furthermore, the subjects include at least 35 subjects, all of whom have at least one year of driving experience or a driving record of 10,000 kilometers or more, and all of whom have normal or corrected-to-normal vision and normal color vision.

[0017] Furthermore, step 2 specifically includes: step 2 specifically includes: the attention model adopts an existing attention model, and obtains two different eye movement scanning schemes of the subject's gaze point and the gaze point predicted by the model by adopting the existing attention model to analyze the preprocessed original driving video.

[0018] Furthermore, step 3 includes four key features related to eye movement scanning, and the key features include: the difference of salient markers between two adjacent frames specified by driving experts, the distance difference between the subject's eye movement scanning point and the model's eye movement scanning point between two adjacent frames, the distance difference between two eye movement scanning schemes in the same frame, and the distance difference between the subject's scanning point and the model's scanning point between the forward and reverse videos, as well as four features of the difference between the optimal salient marker and the suboptimal salient marker.

[0019] Furthermore, the key feature extraction method is as follows:

[0020] Step 4.1: Obtain eye movement data of the subject and the model through eye movement scanning instrument and model prediction;

[0021] Step 4.2: Use wavelet analysis and approximate entropy algorithm to extract eye movement saccades.

[0022] Furthermore, step 5 specifically includes: selecting forward propagation network training, predicting and verifying the remaining preprocessed scan points in the original driving video according to the key features.

[0023] Furthermore, step 6 specifically includes: after obtaining the recommended point plan in step 5, further optimizing the gaze point according to visual comfort and safety, allowing the subjects to drive according to the recommended gaze point, and then participate in judging and scoring.

[0024] It can be seen from the above technical solution that the advantages of the present invention are:

[0025] Compared with the existing technology, in a crowded road environment, especially when the driver's vision is often affected by visual congestion, the present invention selects a more reasonable and better solution by analyzing the key features and scanning trajectories related to eye movements of the driver predicted by the subjects and the model, so as to achieve safe driving and provide a reference for the realization of fully autonomous driving in the future.

[0026] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the attached picture:

[0028] Figure 1 The present invention is a flowchart of a method for recommending safe driving in an urban environment based on the driver's eye movement scanning.

[0029] Figure 2 This is a city driving scene captured by a network camera installed on a mini SUV (Hyundai ix25) according to an embodiment of the present invention.

[0030] Figure 3 This is a highway scene captured by a network camera installed on a mini SUV (Hyundai ix25) according to an embodiment of the present invention.

[0031] Figure 4 This is a driving scene from a city road shot from UE4 (Unreal Engine 4) on the Visual Studio 2019 platform according to an embodiment of the present invention.

[0032] Figure 5 This is a driving scene from a highway shot from UE4 (Unreal Engine 4) on the Visual Studio 2019 platform according to an embodiment of the present invention.

[0033] Figure 6 Static eye movement scatter plot of a driving video clip after applying T-SNE for an embodiment of the present invention.

[0034] Figure 7 A scatter plot of dynamic eye movements in a driving video clip after applying T-SNE for an embodiment of the present invention.

[0035] Figure 8 This is an eye movement scanning clustering diagram of an embodiment of the present invention in a high-speed road driving task.

[0036] Fig. 9 This is an eye movement scanning clustering diagram of an embodiment of the present invention in a driving task on a road in an urban environment. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] See also Figures 1 to 9 The present invention discloses a safe driving recommendation method in an urban environment based on the driver's eye movement scanning, comprising the following steps:

[0039] Step 1: Raw driving video acquisition and preprocessing,

[0040] Recruit subjects with driving experience to participate in the eye movement experiment, collect original driving videos of virtual and real urban driving scenes, and pre-process the original driving videos;

[0041] Step 2: Obtain the eye movement trajectory and fixation point of the subject and the model.

[0042] The common eye movement saccade trajectory of the subjects is obtained through an eye movement saccade device; a representative attention model is selected to predict the fixation point and corresponding saccade trajectory of the original driving video after preprocessing;

[0043] Step 3: Preliminary selection of scanning trajectory scheme,

[0044] By comparing the differences in fixation points and saccades between the subject and the model, the optimal solution for the saccade trajectory is preliminarily selected;

[0045] Step 4: Extract key features related to saccadic eye movements from the pre-processed raw driving video;

[0046] Step 5: Build a neural network model and select the most reasonable fixation point.

[0047] Build a neural network model and select the most reasonable fixation point by referring to the overall safe driving plan;

[0048] Step 6: Optimize the focus point of step 5 and determine the final safe driving plan.

[0049] Optimize the fixation points selected in step 5 to improve driving comfort, involve subjects through a subjective evaluation scheme, and ultimately promote the goal of safe driving.

[0050] Specifically, during the implementation of the present invention, a small passenger car (with less than 7 seats) and a high-definition camera are used to record external driving scenes.

[0051] S1: Collect and pre-process as many comprehensive and rich virtual and real driving scenes as possible, such as highways, urban roads, freeways, and natural scenes such as rainy and foggy days;

[0052] S2: Obtain the subject's eye movement data through the eye tracker, and obtain the predicted eye movement data through the attention model;

[0053] S3: By analyzing the differences between the subjects' eye movements and predicted eye movements;

[0054] S4: Extract multi-dimensional key features from the pre-processed original driving video;

[0055] S5: Train the neural network to determine the gaze point in different driving scenarios, and recommend safe driving solutions based on the opinions of driving experts (ground truth) and verify their rationality.

[0056] S6: Optimize the selection of gaze points, improve comfort to further improve safety, and finally determine and recommend safe driving plans.

[0057] like Figures 2 to 5As shown in the figure, the experiment needs to configure a small passenger car (less than 7 seats) with a wide foreground field of view when collecting data. At the same time, the original driving video database should collect all driving tasks encountered by drivers in real life as much as possible, such as: virtual and real urban driving scenes include complex and simple highways and urban road driving scenes, and emphasize different weather conditions. All original driving videos need to consider both left-hand drive (such as the UK or Australia) and right-hand drive (such as China or the United States) scenes. Half of the original driving videos were shot under bad weather conditions, namely rain, fog, snow, strong wind and night, and the other half were shot on sunny days. Six high-definition color video clips were extracted from 10 different driving environments and recorded with a digital camera mounted on the car windshield. In addition, the video was shot with a Panasonic HX-DC3 high-definition camera with a resolution of 640×480 pixels and 30 frames per second. The high-definition camera was fixed on a tripod to ensure good image quality. .

[0058] Specifically, Figure 2 It is a network camera installed on a mini SUV to collect urban driving picture information. Figure 3 It is highway image information collected by a network camera installed on a mini SUV. Figure 4 This is a driving scene of a city road shot by UE4 on a virtual platform. Figure 5 This is a driving scene of a highway shot by UE4 on a virtual platform. Driving pictures in different scenes were collected to ensure the extensiveness of the collected driving pictures.

[0059] At least 35 subjects, adults (7 females, 28 males) aged between 21 and 42 years old, with an average age of 32.4 ± 0.42 years (mean ± SEM), are required to voluntarily participate in this study. All participants have at least one year of driving experience or a driving record of 10,000 kilometers or more. All subjects have normal or corrected-to-normal vision and normal color vision. At the same time, the existing attention model is used to analyze the above original driving videos to obtain 2 different schemes, namely the subject's gaze point and the model-predicted gaze point. .

[0060] The preprocessing of the original driving video includes color thresholding and Gaussian filtering of the original driving video.

[0061] Specifically, the device for collecting the eye movements of the subject is an eye movement scanning device, and the eye movement scanning device adopts the eye tracker under the Sunlight Eye Movement Platform 2.0.

[0062] The attention model adopts the classic attention model. The preprocessed original driving video is analyzed using the classic attention model to obtain the subject's gaze point and the gaze point predicted by the model.

[0063] Four key eye movement saccadic features are analyzed and extracted, namely (1) the difference in salient markers between two adjacent frames specified by driving experts, (2) the distance difference between the subject's eye movement saccadic points and the model's eye movement saccadic points between two adjacent frames, (3) the distance difference between two eye movement saccadic schemes in the same frame and the distance difference between the subject's saccadic points and the model's saccadic points between the forward and reverse videos, and (4) the difference between the optimal salient marker and the suboptimal salient marker.

[0064] Specifically, the key features related to eye movement and scanning are extracted as follows:

[0065] Step a: obtaining the eye movement data of the subject through an eye movement scanner;

[0066] Step b: Wavelet analysis and approximate entropy algorithms are used to extract key features related to eye movements. The forward propagation network is selected to train, predict and verify the remaining preprocessed driving videos. The visualization effect of high-dimensional classification of gaze point data in virtual and real driving scenes is shown in the figure. Figure 6-7 As shown.

[0067] Specifically, Figure 6 As shown in FIG. 1 , the eye movement scatter plot of the driving video clip collected after T-SNE is applied in this embodiment. Among them: blue (1) represents the eye movement scatter plot on other static visual stimuli, red (2) represents the eye movement scatter plot on the traffic light, and green (3) represents the eye movement scatter plot on the traffic sign. Figure 7 As shown in FIG. 1 , another eye movement scatter plot of the driving video collected after T-SNE is applied in this embodiment. Green (4) represents eye movement scatter plots for pedestrian dynamic stimulation, (5) blue represents eye movement scatter plots for other dynamic visual stimulation, and (6) red represents eye movement scatter plots for moving vehicle visual stimulation. By collecting static and dynamic eye movement scatter plots, a more accurate basis is provided for the selection of safe driving solutions.

[0068] After obtaining the fixation point solution, the fixation point is further optimized according to visual comfort. The subjects drive according to the fixation point and make judgments and scores, and finally determine the recommended safe driving solution. Figures 8 to 9 The cluster diagram of eye movement saccades in different environments is shown. Image 8 to Fig. 9 The far right of the figure shows the eye movement clustering diagram after using the visual safety method and the recommended solution.

[0069] Specifically, Figure 8As shown in the figure, in the high-speed road driving task, the red circle area (7) represents the eye movement clustering of the novice driver; the green circle area (8) represents the eye movement clustering within the visual safety range; the black circle area (9) represents the eye movement clustering after adopting both the visual safety range and the recommended strategy. Fig. 9 As shown in the figure, in the urban environment road task, the red circular area (10) is the eye movement clustering of novice drivers; the green circular area (11) is the eye movement clustering within the visual safety range; the black circular area (12) represents the eye movement clustering after using both the visual safety range and the recommended strategy. The recommended eye movement scheme is finally determined by comprehensively evaluating and scoring novice drivers, visual safety ranges, and the two in different environments.

[0070] First, the original driving video is collected, and the attention model is used to locate the gaze point in the original driving video. Secondly, the eye movement saccade is used to measure the subject's recommended saccades, and the delay between the eye movement saccade trajectory predicted by the model and the subject's saccade trajectory under different driving conditions is analyzed, as well as the delay between the model-predicted saccade trajectory and the subject's controlled saccade trajectory. The visual safety distance is measured by the total delay, and then the preliminary plan is determined. Then, by extracting the four key features of eye movement saccades on the original driving video, the gaze point is optimized, and the final recommended safe driving plan is determined, which can provide reasonable recommendations for future autonomous driving.

[0071] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A safe driving recommendation method in an urban environment based on driver eye movement scanning, characterized in that: The following steps are involved: Step 1: Collection and preprocessing of original driving videos: collecting original driving videos of virtual and real urban driving scenes, and preprocessing the original driving videos; Step 2: Obtain the eye movement trajectories and fixation points of the subjects and the model, recruit subjects with driving experience to participate in the eye movement experiment, obtain the common eye movement trajectories of the subjects, and obtain the common eye movement trajectories of the subjects through the eye movement saccade device; select an attention model to predict the fixation points and corresponding saccade trajectories of the preprocessed driving video; Step 3: Preliminary selection of the scanning trajectory scheme. By comparing the differences in the fixation points and scanning of the subjects and the model, the optimal scanning trajectory scheme is preliminarily selected; Step 4: Extract key features related to eye movements in the preprocessed raw driving video; Step 5: Build a neural network model and select the most reasonable fixation point. Build a neural network model and select the most reasonable fixation point by referring to the overall safe driving plan to meet the requirements of safe driving. Step 6: Optimize the fixation point in step 5 and determine the final recommended safe driving plan.

2. The safe driving recommendation method in an urban environment based on the driver's eye movement scanning as claimed in claim 1, characterized in that: The acquisition of original driving videos for virtual and real urban driving scenes includes: half of the original driving videos are shot in real scenes, and the other half of the original driving videos are shot in virtual scenes, the videos in real scenes in the original driving videos are shot with a digital camera installed on the windshield of the car, and the videos in virtual scenes in the original driving videos are shot with a high-definition camera installed on a tripod.

3. The safe driving recommendation method in an urban environment based on the driver's eye movement scanning as claimed in claim 2, characterized in that: The digital camera installed on the car windshield records driving videos of 10 different driving environments in real scenes, and extracts driving video clips corresponding to 6 different driving environments therefrom. The high-definition camera installed on the tripod records driving videos of 10 different driving environments in virtual scenes, and extracts driving video clips corresponding to 6 different driving environments therefrom.

4. The safe driving recommendation method in an urban environment based on the driver's eye movement scanning as claimed in claim 3, characterized in that: The preprocessing of the original driving video includes performing color threshold processing and Gaussian filtering processing on the original driving video.

5. The safe driving recommendation method in an urban environment based on the driver's eye movement scanning as claimed in claim 4, characterized in that: Step 2 specifically includes: the subjects include at least 35 subjects, all of whom have at least one year of driving experience or a driving record of 10,000 kilometers or more, and all of whom have normal or corrected-to-normal vision and normal color vision.

6. The safe driving recommendation method in an urban environment based on the driver's eye movement scanning as claimed in claim 1, characterized in that: Step 2 specifically includes: the attention model adopts an existing attention model, and obtains two different eye movement scanning schemes of the subject's gaze point and the gaze point predicted by the model by adopting the existing attention model to analyze the preprocessed original driving video.

7. The safe driving recommendation method in an urban environment based on the driver's eye movement scanning as claimed in claim 1, characterized in that: Step 3 includes four key features related to eye movement scanning, which include: the difference of salient markers between two adjacent frames specified by driving experts, the distance difference between the subject's eye movement scanning point and the model's eye movement scanning point between two adjacent frames, the distance difference between two eye movement scanning schemes in the same frame, the distance difference between the subject's scanning point and the model's scanning point between the forward and reverse videos, and the difference between the optimal salient marker and the suboptimal salient marker.

8. The safe driving recommendation method in an urban environment based on the driver's eye movement scanning as claimed in claim 1, characterized in that: The key feature extraction method is as follows: Step 4.1: Obtain eye movement data of the subject and the model through eye movement scanning instrument and model prediction; Step 4.2: Use wavelet analysis and approximate entropy algorithm to extract eye movement saccades.

9. The safe driving recommendation method in an urban environment based on the driver's eye movement scanning as claimed in claim 1, characterized in that: Step 5 specifically includes: selecting forward propagation network training, predicting and verifying the remaining preprocessed scan points in the original driving video according to the key features.

10. The safe driving recommendation method in an urban environment based on driver eye movement scanning as claimed in claim 1, characterized in that: Step 6 specifically includes: after obtaining the recommended point plan in step 5, further optimizing the gaze point according to visual comfort and safety, allowing the subjects to drive according to the recommended gaze point, and then participate in the evaluation and scoring.

Images