A vehicle driving assistance method based on eye movement data

Abstract

The application belongs to the technical field of automobile driving assistance, and particularly relates to an automobile driving assistance method based on eye movement data. The eye movement data of experienced drivers in the driving process is collected, the eye movement intention of the drivers in the driving process is analyzed, intelligent analysis is realized through a deep learning network, ordinary drivers are assisted in driving vehicles, driving assistance can be provided for the drivers in the driving process, timely warning can be provided when unexpected situations occur, and the method can assist instructors and college driving training in driving school training.

Description

Technical Field

[0001] This invention belongs to the field of automotive driving assistance technology, and particularly relates to an automotive driving assistance method based on eye-tracking data. Background Technology

[0002] Intelligent vehicles utilize knowledge from numerous disciplines, such as sensor technology, computer decision-making technology, artificial intelligence technology, and automatic control technology, making them a high-tech complex integrating multiple technologies. Currently, research on intelligent vehicles focuses on improving vehicle safety, comfort, and enhancing the driving experience. Intelligent driving is essentially about attracting and diverting attention through cognitive processing. It captures auditory and visual signals through onboard sensors and controls corresponding responsive systems through cognitive computing.

[0003] Current research on driver assistance systems mainly focuses on lane keeping, automatic parking, and brake assist. Intelligent driver assistance systems are still in their early stages, primarily focusing on using cameras to capture information about the vehicle's surroundings and providing timely warnings and alerts when the vehicle deviates from its lane, thereby improving driver safety and ease of operation.

[0004] However, currently, novice drivers often lack a comprehensive understanding of the vehicle's overall condition during driving. Over-reliance on automated driving assistance systems can hinder the improvement of their skills. Furthermore, driving schools often conduct simulated lessons specifically for driving tests, failing to capture the instructor's reactions to various complex situations in a timely manner, thus preventing students from improving their driving skills through learning from their instructors. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a car driving assistance method based on eye-tracking data. This method collects the driver's eye-tracking data to provide driving suggestions and offers warnings in case of emergencies.

[0006] To achieve the above objectives, the specific process of implementing automotive driving assistance according to the present invention is as follows:

[0007] S1. Eye movement data collection during car driving: Eye movement data of driving school instructors during car driving is collected using eye movement data collection equipment;

[0008] S2. Gaze generation based on contrast information: Input the images from the eye movement data collected in step S1 into the contrast information gaze generation network to extract contrast features;

[0009] S3. Motion-based gaze generation: Input three consecutive frames of motion information from the eye-tracking data collected in step S1 into the motion information main gaze generation network, and use the method of assembling multiple 3D modules to perceive motion features and obtain motion features;

[0010] S4. Spatiotemporal feature integration based on motion feature perception: Motion features are treated as prior information, and the consistency between motion features and spatiotemporal features is calculated before the motion features are fused into the contrast features.

[0011] S5. Viewpoint constraints based on cross-layer perceptual interaction of features: By introducing a cross-layer perceptual interaction module of features, the most discriminative features of contrast features and motion features are mined out respectively, and the enhanced contrast features and motion features are obtained.

[0012] S6. Spatial Feature-Sensitive Cross-Model Interaction: After obtaining the enhanced motion and contrast features, an attention-based fusion mechanism is used. First, the common constraints of the contrast and motion features are explored to strengthen the consistency between them. Overemphasizing consistency will lead to the loss of personalized features, so personalized prediction is introduced to obtain the fused features.

[0013] S7. Continuous learning based on eye-tracking dataset expansion: After obtaining the fused features, the network knowledge is transferred by alternating training with knowledge distillation, so as to ensure that the previously learned knowledge can be retained while new knowledge is learned during the learning process, and thus obtain the predicted driving assistance viewpoint data.

[0014] S8. Driving assistance based on eye-tracking data: Step S7 The driving assistance viewpoint data can provide driving suggestions during the driver's driving process. During the driver's driving process, the eye-tracking data model compares the collected viewpoint data with its own predicted viewpoint data to determine whether the driver has noticed the corresponding area. If there is a difference between the driver's gaze data and the predicted viewpoint data, the vehicle will issue a voice prompt to indicate that there is a sudden situation in the current driving direction, thereby improving the driver's awareness of the existence of the sudden situation and assisting the driver in driving.

[0015] S9. Network Training and Testing: Using the dataset collected in step S1 as the training set, which serves as the fitting data for the driving assistance network, the video data collected during driving is first converted into 2D images. These images are then input into the driving assistance network. The difference between the predicted viewpoint data and the coach's viewpoint data is calculated using the Bceloss function. After the network converges, the model is output to a local folder. During network testing, the video data generated during driving is input into the automatic cruise control system with driving assistance to generate viewpoint data. This data is then compared with the actual collected viewpoint data to calculate the degree of consistency between the two. This consistency level provides assistance to the driver's driving process.

[0016] As a further technical solution of the present invention, the contrast information gaze generation network in step S2 adopts a deep learning network, and the extracted contrast features are F. RGB =DeepNet RGB (I RGB ), where DeepNet RGB The detailed definition of (·) is as follows:

[0017]

[0018] Where I RGB DeepNet represents the image information input to the contrast-enhancing network for gaze generation. RGB (·) represents the contrastive information gaze generation network, and ASPP represents the extraction of multi-scale information, with scale information of (2,4,6). F represents the output feature of the i-th convolutional layer ConvBlock. RGB_i .

[0019] As a further technical solution of the present invention, the motion information gaze generation network in step S3 adopts a deep learning network to obtain motion features. Among them, DeepNet Flow The detailed definition of (·) is as follows:

[0020]

[0021] Among them, DeepNet Flow (·) represents the motion information gaze generation network; These represent three consecutive frames from which motion information is extracted. F represents the output feature of the i-th 3DBlock. Flow_i .

[0022] As a further technical solution of the present invention, the process of calculating the consistency between motion features and spatiotemporal features in step S4 is as follows:

[0023]

[0024] Among them, Cons i,j The consistency value at position (i,j) of the H×W matrix is ​​represented by W and H, which represent the width and height of the local region feature, respectively.

[0025] As a further technical solution of the present invention, the process of obtaining the enhanced contrast feature in step S5 is as follows:

[0026]

[0027]

[0028] The process of obtaining the enhanced motion features is as follows:

[0029]

[0030]

[0031]

[0032] in, and The representative will F RGB and F Flow And the corresponding enhancement features are transformed into different feature spaces; Ψ represents the feature overlay operation; Ψ represents reducing the dimensionality of the feature layer. Represents a linear transformation layer; F represents the normalization function; RGB "and F Flow "" represents the enhanced contrast and motion features.

[0033] As a further technical solution of the present invention, the fusion feature obtained in step S6 is as follows:

[0034]

[0035] Here, Fuse represents the final fusion feature.

[0036] As a further technical solution of the present invention, the process of step S7 is expressed as follows:

[0037]

[0038] Where DeepNet represents DeepNet RGB (·) and DeepNet Flow The combination of (·); Video FixationThe video data collected includes eye-tracking data. The content learned by the Teacher network will be transferred to the Student network in the form of knowledge transfer. When new driving data is available, the Student network will act as the Teacher to guide the Teacher network.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] (1) It can provide driving assistance during the driving process. By collecting the difference between the driver's eye movement data and the eye movement data of experienced drivers, it can assist the driver in driving, provide driving suggestions to the driver, and provide timely warnings in case of emergencies.

[0041] (2) Assisting instructors and students in driving school training can provide convenience for driving school training. Through driving assistance methods, students can improve their skills, and instructors can teach according to students' aptitude. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the car driving assistance method based on eye-tracking data provided by the present invention.

[0043] Figure 2 This is a network structure diagram used in the present invention to realize car driving assistance based on eye-tracking data. Detailed Implementation

[0044] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0045] Example:

[0046] like Figure 1 As shown in Figure 2, this embodiment provides a car driving assistance method based on eye-tracking data, and the specific implementation includes the following steps:

[0047] S1. Eye-tracking data collection during vehicle driving

[0048] To collect eye-tracking data during driving, this embodiment uses the Tobii ProGlasses2 head-mounted eye tracker as the eye-tracking data collection device. It collects eye-tracking data from driving instructors and stores the data locally. This embodiment selects 10 driving instructors and collects data in three scenarios: city center, city outskirts, and city highways. To ensure data accuracy and increase data volume, this embodiment adopts a "multiple-person, multiple-time" collection method. That is, the video data collected by the instructors during driving is demonstrated indoors to other instructors, and their eye-tracking data is collected when they encounter the same situation. Therefore, each collected video segment will contain eye-tracking data from multiple instructors.

[0049] S2. Gaze generation based on contrast information

[0050] During driving, drivers are often exposed to various unexpected situations that distract them. These unexpected situations are primarily objects with significant contrast to the background, such as pedestrians wearing brightly colored clothing on the roadside or vehicles that stand out sharply from their surroundings. Deep learning networks are more sensitive to strong contrast information and can detect areas of significant contrast, thus obtaining the contrast feature F. RGB =DeepNet RGB (I RGB ), where DeepNet RGB The detailed definition of (·) is as follows:

[0051]

[0052] Where I RGB DeepNet represents the image information input into a deep learning network. RGB (·) represents the contrastive information gaze generation network, and ASPP represents the extraction of multi-scale information, with scale information of (2,4,6). F represents the output feature of the i-th convolutional layer ConvBlock. RGB_i .

[0053] S3. Gaze generation based on motion information

[0054] During driving, drivers are more interested in moving objects because these objects are often accompanied by events, such as the movement of vehicles and pedestrians, and changes in traffic lights. This motion information can also be used to extract motion features and detect corresponding moving regions through deep learning networks. Currently, motion information perception is mainly achieved through optical flow extraction. However, optical flow extraction has significant drawbacks, namely, it consumes a large amount of computing time and memory. Therefore, this embodiment uses the assembly of multiple 3D modules to perceive motion features and obtain the motion characteristics. Among them, DeepNet Flow The detailed definition of (·) is as follows:

[0055]

[0056] Among them, DeepNet Flow (·) represents the motion information gaze generation network; Three consecutive frames representing the extraction of motion information; F represents the output feature of the i-th 3DBlock. Flow_i .

[0057] S4. Spatiotemporal feature integration based on motion feature perception

[0058] The above steps yield motion features. The next step is to fuse these motion features to obtain the final output. While motion features are crucial for final viewpoint perception, they are not always effective for moving objects. When an object becomes stationary, motion feature perception fails. Directly fusing motion features with contrast features can degrade their quality. To address this issue, this embodiment treats motion features as prior information instead of directly fusing them with contrast features. It calculates the consistency between motion features and spatiotemporal features before fusing them with the contrast features. The detailed implementation of this consistency is as follows.

[0059]

[0060] Among them, Cons i,j The consistency value at position (i,j) of the H×W matrix represents the value of the local region feature, where W and H represent the width and height of the feature, respectively; F is calculated... RGB and F Flow Pixel errors in local areas can reduce the response values ​​of inconsistent regions in the features of the two, thereby suppressing the final detection error caused by false detection.

[0061] S5. Viewpoint Constraints Based on Feature-Based Cross-Layer Perceptual Interaction

[0062] The most critical issue for drivers is identifying the most discriminative regions during driving. Meanwhile, human perception of objects often follows a coarse-to-fine process. However, relying solely on traditional overlay modules can lead to the loss of local details between layers and the introduction of global ambiguity. Therefore, this embodiment introduces a cross-layer feature perception interaction module to adaptively extract the most discriminative features. The flow process of the contrastive features is as follows:

[0063]

[0064]

[0065] The detailed flow process of the motion characteristics is as follows:

[0066]

[0067]

[0068] in, and The representative will F RGB and F FlowAnd the corresponding enhancement features are transformed into different feature spaces; Ψ represents the feature overlay operation; Ψ represents reducing the dimensionality of the feature layer. Represents a linear transformation layer; F represents the normalization function; RGB "and F Flow "" represents the final output feature.

[0069] S6. Spatial Feature-Sensitive Cross-Model Interaction

[0070] After obtaining the enhanced motion and contrast features through the above methods, it is necessary to fully explore the complementary features between them to leverage their respective advantages. Unlike traditional fusion methods that rely on superposition, this embodiment employs an attention-based fusion mechanism. This involves first identifying the common constraints between the two features to strengthen their consistency. However, overemphasizing consistency can lead to the loss of personalized features. Therefore, this embodiment introduces personalized prediction based on the exploration of common constraints, the detailed definition of which is as follows.

[0071]

[0072] Here, Fuse represents the final fusion feature; the definitions of other symbols are as in step S7.

[0073] S7. Continuous learning based on eye-tracking dataset augmentation

[0074] While the eye-tracking data collected by driving instructors using the methods described above can meet the training requirements of the model, the constantly changing environment allows instructors to continue collecting data based on the driving assistance system. Furthermore, the discrepancy between the deep learning model's predictions and the instructor's own data can help identify current model deficiencies. Therefore, it is necessary to design a deep learning model that is not static but can continuously learn and improve its performance in response to environmental changes. Although fine-tuning can address these issues, it can lead to "catastrophic forgetting" problems. Therefore, this embodiment employs an alternating training method of knowledge distillation to transfer network knowledge, ensuring that previously learned knowledge is retained while new knowledge is acquired during the learning process, thus obtaining driving assistance viewpoint information. Specifically:

[0075]

[0076] Where DeepNet represents DeepNet RGB (·) and DeepNet Flow The combination of (·); Video FixationThe video data represents collected eye-tracking data. The content learned by the Teacher network will be transferred to the Student network in the form of knowledge transfer. When there is new driving data, the Student network will act as the Teacher to guide the Teacher network (Student). Knowledge distillation adopts the Teacher-Student model: the complex and large model is used as the Teacher, and the Student model has a simpler structure. The Teacher is used to assist the training of the Student model. The Teacher has a strong learning ability and can transfer the knowledge it has learned to the Student model with a relatively weak learning ability, thereby enhancing the generalization ability of the Student model.

[0077] S8, Eye-tracking data-based driving assistance

[0078] The driver assistance viewpoint information generated in the above manner can provide driving suggestions during the driver's driving process. During driving, the eye-tracking data model compares the collected viewpoint data with its own predicted viewpoint data to determine whether the driver is currently noticing the corresponding area. If there is a discrepancy between the driver's gaze data and the viewpoint area predicted by the deep learning model, the vehicle will issue a voice prompt indicating a sudden situation in the current driving direction, thereby improving the driver's awareness of the sudden situation and assisting the driver in driving. Specifically:

[0079] Peo=‖max{T-abs(Binary(EP Real )-Binary(DeepNet(Video Fixation )))}‖0 / θ

[0080] Where, Peo indicates whether to issue a voice alarm; T(30%) represents the threshold, that is, the balance point of whether an alarm reminder is needed. In this embodiment, a percentage is used. Since a fixed threshold cannot change with the viewpoint, a percentage is used as the threshold; Binary indicates binarizing the viewpoint; abs is used to calculate the degree of consistency between viewpoints.

[0081] S9, Network Training

[0082] The dataset collected in step S1 is used as the training set, which is then used as the fitting data for the driving assistance network. First, the video data collected during the driving process is converted into 2D images (512*512), and the image data during the driving process is input into the driving assistance network. The difference between the predicted viewpoint data and the coach's viewpoint data is calculated by the Bceloss function. After the network converges (12 epoach), the network model is output to the local folder.

[0083] S10, Network Testing

[0084] Video data generated during driving is input into an automated cruise control system with driver assistance features to generate viewpoint data fixation. This fixation is then compared with the actual viewpoint data collected to calculate the degree of consistency between the two. The degree of consistency is used to assist the driver's driving process.

[0085] This embodiment utilizes a deep learning model, which has been continuously developed due to its ability to learn and simulate human capabilities. Step S1 yields a large amount of street video information with viewpoint details. This embodiment collects eye-tracking data from "experienced" drivers during driving to analyze their behavioral intentions. Compared to braking and driving, analyzing the driver's eye-tracking intentions during driving is more effective in revealing how "experienced" drivers (i.e., instructors) handle unexpected situations. This allows for intelligent analysis through a deep learning network, assisting ordinary drivers in driving. The entire process can be represented as follows:

[0086]

[0087] Among them, EP Real and EP Past These represent real eye-tracking data and past eye-tracking data, respectively; DeepNet RGB (·) and ΘDeepNet Flow (·) represent the contrast information gaze generation network and the motion information gaze generation network, respectively; I RGB This represents the image information input into the deep learning network; Represents three consecutive frames from which motion information is extracted; DeepNet represents DeepNet. RGB (·) and ΘDeepNet Flow The combination of (·); Video Fixation ← represents the collected video data with eye-tracking data; ← represents the flow of information between data; Θ represents the feature interaction module.

[0088] The eye-tracking data model generated in this embodiment can be used not only for driver assistance systems but also for driving school student training. Driving schools often rely on instructors to train students based on the requirements of the driving school exam. However, this method of training is exam-oriented and lacks real-world practicality. With the driver assistance system proposed in this patent, students can directly learn the instructor's driving style, thus directly learning to drive in real-world situations. This also greatly simplifies the instructor's training process and reduces their workload. Furthermore, driving schools can collect students' eye-tracking data and compare it with the instructor's data to promptly identify driving problems during the student's driving process, allowing for targeted and effective instruction, significantly facilitating teaching.

[0089] Algorithms, network structures, and models not described in detail in this article are all general techniques in this field.

[0090] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A car driving assistance method based on eye-tracking data, characterized in that, The specific process of implementing car driver assistance is as follows: S1. Eye movement data collection during car driving: Eye movement data of driving school instructors during car driving is collected using eye movement data collection equipment; S2. Gaze generation based on contrast information: Input the images from the eye movement data collected in step S1 into the contrast information gaze generation network to extract contrast features; S3. Motion-based gaze generation: Input three consecutive frames of motion information from the eye-tracking data collected in step S1 into the motion information gaze generation network, and use the method of assembling multiple 3D modules to perceive motion features and obtain motion features. S4. Spatiotemporal feature integration based on motion feature perception: Motion features are treated as prior information, and the consistency between motion features and spatiotemporal features is calculated before the motion features are fused into the contrast features. S5. Viewpoint constraints based on cross-layer perceptual interaction of features: By introducing a cross-layer perceptual interaction module of features, the most discriminative features of contrast features and motion features are mined out respectively, and the enhanced contrast features and motion features are obtained. S6. Spatial Feature-Sensitive Cross-Model Interaction: After obtaining the enhanced motion and contrast features, an attention-based fusion mechanism is used. First, the common constraints of the contrast and motion features are explored to strengthen the consistency between them. Overemphasizing consistency will lead to the loss of personalized features, so personalized prediction is introduced to obtain the fused features. S7. Continuous learning based on eye-tracking dataset expansion: After obtaining the fused features, the network knowledge is transferred by alternating training with knowledge distillation, so as to ensure that the previously learned knowledge can be retained while new knowledge is learned during the learning process, and thus obtain the predicted driving assistance viewpoint data. S8. Driving assistance based on eye-tracking data: Step S7 The driving assistance viewpoint data can provide driving suggestions during the driver's driving process. During the driver's driving process, the eye-tracking data model compares the collected viewpoint data with its own predicted viewpoint data to determine whether the driver has noticed the corresponding area. If there is a difference between the driver's gaze data and the predicted viewpoint data, the vehicle will issue a voice prompt to indicate that there is a sudden situation in the current driving direction, thereby improving the driver's awareness of the existence of the sudden situation and assisting the driver in driving. S9. Network Training and Testing: Using the dataset collected in step S1 as the training set, which serves as the fitting data for the driving assistance network, the video data collected during driving is first converted into 2D images. These images are then input into the driving assistance network. The difference between the predicted viewpoint data and the coach's viewpoint data is calculated using the Bceloss function. After the network converges, the model is output to a local folder. During network testing, the video data generated during driving is input into the automatic cruise control system with driving assistance to generate viewpoint data. This data is then compared with the actual collected viewpoint data to calculate the degree of consistency between the two. This consistency level provides assistance to the driver's driving process.

2. The vehicle driving assistance method based on eye-tracking data according to claim 1, characterized in that, The contrast information gaze generation network in step S2 employs a deep learning network, and the extracted contrast features are: ,in The detailed definition is as follows: ， in This represents the image information input into the contrast-information gaze generation network. The network represents the contrastive information gaze generation network, and ASPP represents the extraction of multi-scale information with scale information of (2,4,6). Representing the Convolutional layers Output characteristics .

3. The vehicle driving assistance method based on eye-tracking data according to claim 2, characterized in that, The motion information gaze generation network in step S3 uses a deep learning network to obtain motion features. ,in, The detailed definition is as follows: ， in, A gaze-generating network representing motion information; These represent three consecutive frames from which motion information is extracted. Representing the indivual Output characteristics .

4. The vehicle driving assistance method based on eye-tracking data according to claim 3, characterized in that, Step S4, calculating the consistency between motion features and spatiotemporal features, is as follows: ， in, The consistency value at position (i,j) of the H×W matrix is ​​represented by W and H, which represent the width and height of the local region feature, respectively.

5. The vehicle driving assistance method based on eye-tracking data according to claim 4, characterized in that, The process of obtaining the enhanced contrast features in step S5 is as follows: ， ， The process of obtaining the enhanced motion features is as follows: ， ， in, , and , The representative will and And the corresponding enhancement features are transformed into different feature spaces; Represents the feature overlay operation; This represents reducing the dimensionality of the feature layer; Represents a linear transformation layer; Represents the normalization function; This represents the enhanced contrast and motion features.

6. The vehicle driving assistance method based on eye-tracking data according to claim 5, characterized in that, The fusion features obtained in step S6 are: ， Here, Fuse represents the final fusion feature.

7. The vehicle driving assistance method based on eye-tracking data according to claim 6, characterized in that, Step S7 is represented as follows: ， in, represent and The combination; This represents the collected video data containing eye-tracking data. The content representing online learning will be transferred through knowledge transfer. When the network has new driving data, The internet will serve as guide network.

Images