Driving distraction state recognition method based on analysis of spatiotemporal semantic features of eye movement sequences

Abstract

The application discloses a driving distraction state recognition method based on eye movement sequence space-time semantic feature analysis, and relates to the technical field of visual analysis, and comprises the following steps: S1, acquiring eye movement data of a to-be-identified subject during driving; S2, constructing a driving distraction recognition model based on eye movement sequence space-time semantic features; and S3, inputting the eye movement data of the to-be-identified subject into the driving distraction recognition model based on eye movement sequence space-time semantic features which is constructed, and obtaining a distraction detection result of the to-be-identified subject. The application designs a driving distraction recognition model based on eye movement sequence space-time semantic features, extracts time, space and semantic features in visual saccade paths, and combines the features to represent the eye movement mode of a driver, so that whether the driver is in a distraction state can be identified.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual analysis, and more particularly to a driving distraction state recognition method based on eye movement sequence space-time semantic feature analysis. BACKGROUND

[0002] With the progress of the times, cars have entered thousands of households. The number of traffic accidents has also increased year by year, and the distraction behavior of car drivers during driving has become one of the main reasons for serious traffic accidents. Driving distraction refers to the delay in recognizing the information required for safe driving due to some events, activities, objects or people inside or outside the vehicle. According to the statistics of NHTSA, about 25% of accidents are caused by some form of inattention, such as the driver being distracted by doing other things while driving or being distracted by fatigue driving. In order to reduce the occurrence of traffic accidents, it is essential to develop a monitoring system to detect the inattention and distraction of the driver.

[0003] The method for detecting driver distraction is divided into three categories: mathematical model, rule-based model and machine learning algorithm-based model. At present, most of them use machine learning algorithm model to recognize driver distraction. The input features of the distraction recognition model can be roughly divided into the following categories:

[0004] First, facial, hand and other image features. The image visual data of the driver includes the images or videos of the driver's facial expressions, eye movements and body movements. The model based on image features has the characteristics of strong practicability and low recognition cost. However, due to the constantly changing light environment in the cockpit, there is a great limitation in accurately and timely detecting the driver's behavior. In addition, this method highly depends on the picture texture features extracted manually by experts, otherwise the accuracy cannot meet the practical requirements.

[0005] Second, vehicle dynamics features collected by vehicle sensors. Vehicle dynamics related feature data includes steering wheel rotation angle, lane deviation, vehicle acceleration and other driving performance indicators. The driver's behavior directly affects the performance of the vehicle, and the data of each controller (vehicle speed, steering wheel angle, etc.) are collected and analyzed by the vehicle network to determine whether the driver is in a distracted state. However, due to the variability of driving scenarios, it is difficult to accurately monitor the current state of the driver through structured data. In addition, in the field of autonomous driving, there is no such data.

[0006] Third, physiological data indicators. Physiological data mainly include eye movement, electroencephalogram (EEG), electromyogram (EMG) and electrocardiogram (ECG) and other data. Generally speaking, through the physiological information data recognition driving behavior can ensure that the system accurate and rapid output results. But in addition to eye movement data of other physiological data acquisition, the driver must be equipped with physiological sensor device, such as invasive equipment may cause the driver's driving discomfort, and has certain negative impact on the normal driving behavior. Therefore, eye movement data is more suitable for the research of driving distraction recognition, on this basis, how to accurately and effectively identify whether the driver is in a state of distraction is a technical problem that the skilled in the art need to solve. SUMMARY

[0007] Therefore, the application provides a driving distraction state recognition method based on eye movement sequence space-time semantic feature analysis, which can accurately and effectively identify whether the driver is in a state of distraction.

[0008] In order to achieve the above purpose, the application provides the following technical scheme:

[0009] A driving distraction state recognition method based on eye movement sequence space-time semantic feature analysis, comprising the following steps:

[0010] S1, obtaining the eye movement data of the subject to be identified when driving;

[0011] S2, constructing a driving distraction recognition model based on eye movement sequence space-time semantic feature;

[0012] S3, inputting the eye movement data of the subject to be identified into the driving distraction recognition model based on eye movement sequence space-time semantic feature constructed, to obtain the distraction detection result of the subject to be identified.

[0013] Optionally, the construction of the driving distraction recognition model based on eye movement sequence space-time semantic feature in S2 comprises the following steps:

[0014] S21, obtaining driving simulation experiment scene and distraction scene feature map;

[0015] S22, obtaining the eye movement data of the sample subject when driving; wherein the sample subject includes distracted driving and normal driving;

[0016] S23, obtaining the eye movement sequence according to the eye movement data of the sample subject;

[0017] S24, dividing the interest area of the driving scene according to the eye movement sequence and semantic information;

[0018] S25, obtaining an eye movement sub-sequence by a sliding window according to the eye movement sequence;

[0019] S26, constructing a feature sequence of a fixation point according to the eye movement sub-sequence; wherein the feature sequence comprises a time feature sequence, a space feature sequence and an interest region semantic feature sequence;

[0020] S27, training an XGBoost model constructed by using the feature sequence and a classification label pair of a driving state corresponding to the feature sequence, and classifying an input feature by using the trained XGBoost model to obtain a driving distraction state identification result of a sample subject.

[0021] Optionally, the driving simulation experiment scene and the distraction scene feature map are obtained in S21, and the obtaining specifically comprises:

[0022] The driving simulation experiment scene is built based on a UC-win / Road 14.0 simulation software, a secondary task is designed to induce a driver to be distracted, and a feature map is obtained according to the distraction scene.

[0023] Optionally, the eye movement sequence is obtained in S23, and the obtaining specifically comprises the following steps:

[0024] S231, mapping eye movement data of the sample subject to a coordinate system of the distraction scene feature map to obtain mapped coordinate point data;

[0025] S232, extracting a fixation point based on a speed threshold algorithm on the mapped coordinate point data to obtain horizontal and vertical coordinates of the fixation point and a time stamp formed by the fixation point, and determining the horizontal and vertical coordinates and the time stamp as the eye movement sequence.

[0026] Optionally, the specific steps of S232 are as follows:

[0027] The coordinate points in a set A composed of the mapped coordinate point data are sequentially obtained, and an eye movement speed of the obtained coordinate point to a next coordinate point of the obtained coordinate point is calculated;

[0028] If the eye movement speed is less than a preset speed threshold, the obtained coordinate point is added to a set B, and a next coordinate point is continuously obtained until the eye movement speed of the obtained coordinate point is greater than or equal to the preset speed threshold, whether the number of coordinate points in the set B is greater than or equal to a preset number threshold is judged, if yes, the set B constitutes a fixation, and a fixation point is obtained, if no, the set B does not constitute a fixation, and a next coordinate point is continuously obtained until the coordinate points in the set A are obtained.

[0029] Optionally, the interest region of the driving scene is divided in S24, and the dividing specifically comprises the following steps:

[0030] S241, cluster analysis is performed on the coordinate points in the eye movement sequence to obtain a clustering result;

[0031] S242, according to the semantic information, the clustering result is obtained to obtain the interest region division result.

[0032] Optionally, in S25, the eye movement sub-sequence is obtained by using a sliding window to intercept the eye movement sequence, and the specific process is as follows:

[0033] The eye movement data of the sample subject is intercepted by using a 3s time window and a 1.5s sliding step to obtain the eye movement sub-sequence of the sample subject.

[0034] Optionally, the eye movement sub-sequence includes the horizontal and vertical coordinates of the fixation point, the semantic information of the interest region, and the time stamp.

[0035] Optionally, in S26, the feature sequence of the fixation point is constructed according to the eye movement sub-sequence, and the specific process includes the following steps:

[0036] S261, the eye movement sequence of the fixation point of the preset segment is obtained;

[0037] S262, according to the horizontal and vertical coordinates in the obtained eye movement sequence, the image features of the corresponding positions on the feature map are obtained to obtain the spatial feature sequence, the time feature sequence, and the interest region semantic feature sequence.

[0038] Compared with the prior art, the application provides a driving distraction state recognition method based on eye movement sequence space-time semantic feature analysis, eye movement data of subjects in distraction and non-distraction states is collected through driving simulation experiment. In the driving process, the contents and behavior modes of the driver are closely related to the eye fixation points and the movement of the line of sight of the driver. The driver needs to frequently scan the road and the surrounding area to ensure accurate perception of the driving environment. If the driver fails to pay attention to the correct position in time when driving the vehicle, it may lead to cognitive errors of key information, thereby increasing the probability of traffic accidents. The eye movement patterns of the driver are different in the distraction and non-distraction states, and the eye movement directly reflects the eye movement pattern. The saccade path can be regarded as the performance of the observer's brain intrinsic characteristics. The similarity of the saccade path can be regarded as the symbol of perception or attention bias, and the interest area can reflect the relevance of the attention focus area of the driver to driving. Therefore, the application uses machine learning to establish a connection between the eye movement pattern and the distraction state of the subject, and realizes a more efficient and easy-to-obtain driving distraction state recognition method. Using the driving distraction state recognition method based on eye movement sequence space-time semantic feature analysis disclosed in the application, efficient driving distraction state recognition can be realized. The application overcomes the shortcomings of the image feature recognition method, such as being easily affected by light and having a complex system, the vehicle dynamics feature is not suitable for the automatic driving field, and the collection of other physiological characteristics will interfere with driving. The application can accurately and efficiently recognize the driving distraction state in the field of traffic safety and improve road safety. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, brief introductions will be given to the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only belong to the embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0040] Figure 1 The flowchart of the driving distraction state recognition method based on eye movement sequence space-time semantic feature analysis provided by the application is shown in the figure.

[0041] Figure 2 The construction process schematic diagram of the driving distraction recognition model based on eye movement sequence space-time semantic feature provided by the application is shown in the figure. DETAILED DESCRIPTION

[0042] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0043] The embodiment of the present application discloses a driving distraction state recognition method based on eye movement sequence space-time semantic feature analysis, as shown in Figure 1 The embodiment of the present application discloses a driving distraction state recognition method based on eye movement sequence space-time semantic feature analysis, as shown in

[0044] S1, acquiring eye movement data of a to-be-identified subject during driving; specifically, the eye movement data of the to-be-identified subject can be recorded based on an eye tracker;

[0045] S2, constructing a driving distraction recognition model based on eye movement sequence space-time semantic features;

[0046] S3, inputting the eye movement data of the to-be-identified subject into the constructed driving distraction recognition model based on eye movement sequence space-time semantic features to obtain a distraction detection result of the to-be-identified subject.

[0047] In a feasible implementation, the present application designs a driving simulation experiment for collecting eye movement data in the states of distraction and non-distraction.

[0048] In a feasible implementation, the present application relates to a comparison method of eye movement pattern similarity, which compares saccade paths and attention transfer characteristics to measure the similarity of eye movement patterns.

[0049] The saccade path comparison method measures the similarity of saccade paths through three dimensions of fixation duration, fixation coordinates and fixation order. Before calculating the similarity of saccade paths, it needs to be simplified because the original eye movement signal cannot directly reflect the fixation pattern of the subject. If the original eye movement coordinate recording points are within a given distance or within a given direction threshold of each other, they are iteratively merged into continuous fixation points. This simplification process helps to reduce the complexity of saccade paths while maintaining their spatial and temporal structure. The simplified saccade paths are position-encoded to form eye movement trajectory strings. A substitution matrix and a gap penalty are set, the strings are aligned using the Needleman-Wunsch algorithm to obtain an alignment score, and finally the normalized score is obtained to obtain the final similarity score.

[0050] The attention shift characteristics are measured by the entropy rate and the visual transition matrix fusion method. Before analyzing the attention shift characteristics, the scene needs to be divided into interest regions. The K-means clustering method is adopted to divide the interest regions into the following regions: the road surface region, the instrument panel region, the distraction region, the rearview mirror region and the other region, in combination with semantic information. The entropy rate is used to measure the randomness of the saccade, and the greater the entropy rate, the greater the randomness or the shorter the average fixation duration and the saccade in multiple regions. The visual transition matrix is used to measure and visualize the distribution of the fixation behavior, i.e. the movement of the visual attention in a certain spatial region.

[0051] Further, the present application obtains the eye movement pattern difference by comparing the saccade paths and the attention shift characteristics of different members belonging to the same group (intra-group similarity) and different groups (inter-group similarity). The present application finds that the saccade paths of the distraction group tend to be different, but the saccade order tends to be consistent; and the attention of the distraction group is more transferred between the distraction region and the road region, and the saccade is more free. The present application considers that in this case, the eye movement pattern can well measure the attention bias of the driver. In the absence of any bias, the eyes will move freely to explore the driving scene with equal probability. In the undistracted state, the attention should be biased to the road, rearview mirror and instrument panel regions. However, because the distracted driver is stimulated by something, the attention is transferred to the distraction region. At this time, the saccade path of the distracted driver tends to be different, and the attention is more transferred to the distraction region. Therefore, it is very important to utilize the spatio-temporal semantic features of the eye movement sequence. The present application obtains the eye movement data of the driver according to the pre-built experimental platform, extracts the spatio-temporal and semantic features, and builds a driving distraction recognition model.

[0052] Further, the construction of the driving distraction recognition model based on the spatio-temporal semantic features of the eye movement sequence in S2, as shown in Figure 2 includes the following steps:

[0053] S21, obtaining a driving simulation experiment scene and a distraction scene feature map;

[0054] Specifically, the driving simulation experiment scene is built based on the UC-win / Road14.0 simulation software, the driver is induced to be distracted by designing a sub-task, and the feature map is obtained according to the distraction scene.

[0055] S22, obtaining eye movement data of a sample subject while driving; wherein the sample subject includes a distracted driving and a normal driving.

[0056] S23, obtaining an eye movement sequence according to the eye movement data of the sample subject;

[0057] Further, S23 includes the following steps:

[0058] S231, map the eye movement data of the sample subject into the coordinate system of the distraction scene feature map to obtain mapped coordinate point data;

[0059] S232, extract the fixation point based on the speed threshold algorithm on the mapped coordinate point data to obtain the horizontal and vertical coordinates of the fixation point and the time stamp formed by the fixation point, and determine the horizontal and vertical coordinates and the time stamp as the eye movement sequence.

[0060] Specifically, S232 is: sequentially obtaining the coordinate points in set A composed of the mapped coordinate point data, calculating the eye movement speed of the obtained coordinate point to the next coordinate point of the obtained coordinate point; if the eye movement speed is less than the preset speed threshold, the obtained coordinate point is added to set B, and the next coordinate point is continuously obtained until the eye movement speed of the obtained coordinate point is greater than or equal to the preset speed threshold, whether the number of coordinate points in set B is greater than or equal to the preset number threshold, if yes, set B constitutes a fixation, and the fixation point is obtained; if not, set B does not constitute a fixation, and the next coordinate point is continuously obtained until the coordinate points in set A are obtained.

[0061] In a feasible implementation, the algorithm for clustering the coordinate points based on the speed threshold is designed to obtain the horizontal and vertical coordinates and the time stamp of the fixation point as the eye movement sequence. Wherein, the speed threshold is set to 100° / s, greater than the threshold is saccade, otherwise is fixation.

[0062] By calculating part of the eye movement trajectory data of the subject in a single experiment, multiple fixation point information of the subject can be obtained for subsequent feature extraction and analysis work.

[0063] S24, according to the eye movement sequence and the semantic information, the interest region of the driving scene is divided;

[0064] Specifically, S24 includes the following steps:

[0065] S241, clustering analysis is performed on the coordinate points in the eye movement sequence to obtain a clustering result;

[0066] S242, according to the semantic information, the clustering result is obtained to obtain the interest region division result.

[0067] In a feasible implementation, all coordinate point information is extracted, and a K-means clustering algorithm is used for analysis. Before using the clustering algorithm, the coordinate point information needs to be subjected to a symmetric transformation, because the eye movement coordinates recorded by the device and the coordinates of the feature map are not corresponding. The specific conversion is shown in the following formula (1), wherein a and b are the resolution of the feature map, (x', y') is the original coordinate, and (x, y) is the transformed coordinate.

[0068] x = a - x'y = b - y' (1);

[0069] S25, according to the eye movement sequence, the eye movement sub-sequence is obtained by sliding window interception;

[0070] Specifically, the eye movement data of the sample subjects is intercepted using a 3s time window and a 1.5s sliding step, and the eye movement sub-sequences of the sample subjects are obtained respectively.

[0071] The eye movement sub-sequence includes the horizontal and vertical coordinates of the fixation point, the semantic information of the interest region, and the timestamp.

[0072] S26, according to the eye movement sub-sequence, a feature sequence of the fixation point is constructed; wherein the feature sequence includes a time feature sequence, a space feature sequence, and an interest region semantic feature sequence;

[0073] Specifically, S26 includes the following steps:

[0074] S261, obtaining the eye movement sequence of the fixation point of the preset segment;

[0075] S262, according to the horizontal and vertical coordinates in the obtained eye movement sequence, the image features of the corresponding positions on the feature map are obtained, and the space feature sequence, the time feature sequence, and the interest region semantic feature sequence are obtained.

[0076] In a feasible implementation, the horizontal and vertical coordinates of the fixation point are extracted to obtain the space feature sequence. According to the timestamp corresponding to each fixation point, the time interval between the next fixation point and the fixation point can be obtained, and the fixation duration time feature sequence is obtained. Moreover, according to the position of the fixation point falling in the interest region, the semantic feature sequence of the fixation interest region can be obtained. Finally, the mean and standard deviation of the feature sequence in each sub-sequence are calculated to obtain the final input feature sequence of the model.

[0077] S27, using the feature sequence and the classification label pair of the driving state corresponding to the feature sequence to train the constructed XGBoost model, and using the trained XGBoost model to classify the input features to obtain the sample subject driving distraction state identification result.

[0078] In a feasible implementation, the space, time, and semantic features of the processed eye movement sequence are used as the eye movement pattern classification index, and distraction and non-distraction are used as the classification label. 70% of the data set is randomly selected as training data, an XGBoost classification model is established, and another 30% of the data set is selected as test data (excluding training data) for predicting the distraction level.

[0079] Further, the application measures the eye movement pattern by using saccade path contrast and attention shift characteristics. By analyzing the similarity of visual saccade paths and quantifying the attention shift characteristics, multi-dimensional analysis of eye movement patterns is realized, and the temporal and spatial semantic features of eye movement signals are fully mined and utilized, and the differences between driving distraction and non-distraction eye movement patterns are quantified.

[0080] In the prior art, only the basic features of eye movement signals are used, such as the number of eye movement points, the diameter of the pupil, etc., and for the first time, the temporal and spatial semantic features of eye movement signals are fully mined and utilized, the gaze point coordinates, time length and interest region gaze frequency features are taken as the input of the driving distraction recognition model, and accurate and efficient driving distraction state identification can be realized. The application overcomes the shortcomings of image feature recognition methods being susceptible to light and being complex, vehicle dynamics features not being suitable for automatic driving fields, and other physiological data collection interfering with driving, and can accurately and efficiently realize driving distraction state recognition in the field of traffic safety and improve road safety.

[0081] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0082] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1.A driving distraction state recognition method based on analysis of spatiotemporal semantic features of eye movement sequences, characterized in that, The method comprises the following steps: S1, obtaining eye movement data of a to-be-identified subject during driving; S2, constructing a driving distraction recognition model based on the spatiotemporal semantic features of eye movement sequences; S3, inputting the eye movement data of the to-be-identified subject into the driving distraction recognition model based on the spatiotemporal semantic features of eye movement sequences, to obtain a distraction detection result of the to-be-identified subject; The construction of the driving distraction recognition model based on the spatiotemporal semantic features of eye movement sequences in S2 comprises the following steps: S21, obtaining a driving simulation experiment scene and a distraction scene feature map; S22, obtaining eye movement data of sample subjects during driving; wherein the sample subjects include distracted driving and normal driving; S23, obtaining an eye movement sequence according to the eye movement data of the sample subjects; S24, dividing the driving scene into interest regions according to the eye movement sequence and semantic information; S25, obtaining an eye movement sub-sequence by using a sliding window according to the eye movement sequence; S26, constructing a feature sequence of fixation points according to the eye movement sub-sequence; wherein the feature sequence comprises a time feature sequence, a space feature sequence, and an interest region semantic feature sequence; S27, training an XGBoost model using the feature sequence and the classification label of the driving state corresponding to the feature sequence, and classifying the input features using the trained XGBoost model to obtain a sample subject driving distraction state identification result; In S21, the driving simulation experiment scene and the distraction scene feature map are obtained as follows: A driving simulation experiment scene is built based on a UC-win / Road 14.0 simulation software, a secondary task is designed to induce the driver to be distracted, and a feature map is obtained according to the distraction scene; In S23, the eye movement sequence is obtained, which comprises the following steps: S231, mapping the eye movement data of the sample subjects to the coordinate system of the distraction scene feature map to obtain mapped coordinate point data; S232, extracting fixation points based on a speed threshold algorithm on the mapped coordinate point data to obtain the horizontal and vertical coordinates of the fixation points and the time stamp formed by the fixation points, and determining the horizontal and vertical coordinates and the time stamp as the eye movement sequence; In S24, the driving scene is divided into interest regions, which comprises the following steps: S241, performing cluster analysis on the coordinate points in the eye movement sequence to obtain a cluster result; S242, obtaining an interest region division result according to the semantic information and combining the cluster result; In S25, the eye movement sub-sequence is obtained by using a sliding window according to the eye movement sequence, which is specifically as follows: A 3s time window and a 1.5s sliding step are used to intercept the eye movement data of the sample subjects to obtain the eye movement sub-sequence of the sample subjects; In S26, the feature sequence of the fixation points is constructed according to the eye movement sub-sequence, which comprises the following steps: S261, obtaining the eye movement sequence of the fixation points of a preset segment; S262, obtaining the image features of the corresponding positions on the feature map according to the horizontal and vertical coordinates in the obtained eye movement sequence, to obtain the space feature sequence, the time feature sequence, and the interest region semantic feature sequence. 2.The driving distraction state recognition method based on eye movement sequence spatio-temporal semantic feature analysis of claim 1, characterized in that, The specific steps of S232 are as follows: The coordinate points in the set A composed of the mapped coordinate point data are acquired in sequence, and the eye movement speed of the acquired coordinate point to the next coordinate point of the acquired coordinate point is calculated; If the eye movement speed is less than a preset speed threshold, the acquired coordinate point is added to the set B, and the next coordinate point is continuously acquired until the eye movement speed of the acquired coordinate point is greater than or equal to the preset speed threshold, whether the number of coordinate points in the set B is greater than or equal to a preset number threshold is judged, if yes, the set B constitutes a fixation, and a fixation point is obtained, if no, the set B does not constitute a fixation, and the next coordinate point is continuously acquired until the coordinate points in the set A are acquired. 3.The driving distraction state recognition method based on eye movement sequence spatio-temporal semantic feature analysis of claim 1, characterized in that, The eye movement sub-sequence includes horizontal and vertical coordinates of the fixation point, semantic information of the interest region and a time stamp.

Images