A driver attention area prediction system

Abstract

The present invention discloses a method for predicting a driver's attention area, comprising the following steps: S1: obtaining an original driving video of a driver's attention area, and preprocessing the original driving video to obtain a processed driving video sequence; S2: using a deep learning Keras framework to build a deep learning model, and training the deep learning model; S3: using the trained deep learning model to perform regional prediction on the preprocessed driving video to obtain a driver's attention area prediction result; S4: outputting the driver's attention area prediction result; and also discloses a driver's attention area prediction system, comprising a driving video acquisition and preprocessing module, a model training module, a model application module, and a result output module; the present invention has the ability to perform differentiated training on driving attention in LHT and RHT scenarios, and can accurately predict driving attention by scenario and situation.

Description

Technical Field

[0001] The present invention relates to the technical field of driver attention prediction, and in particular to a driver attention area prediction system. Background Art

[0002] Driving has long been a task that requires processing a large amount of visual information. Human vision plays a vital role in making appropriate decisions for vehicle control. Therefore, a comprehensive understanding of the driver's visual behavior is an important component for improving driving safety.

[0003] In reality, real-life traffic driving is a complex, dynamic, and ever-changing scenario, characterized by a complex interplay of subjective and objective factors that influence a driver's gaze and attention. These include differences in left-handed traffic (LHT) and right-handed traffic (RHT), as well as safety and hazards. Subjective factors include bottom-up visual stimulation, which typically prioritizes attention to the most important and prominent areas of a visual scene; and task-driven, top-down visual attention mechanisms, which allocate attention to the scene under conscious control, incorporating prior knowledge. While driving, drivers typically focus their attention on the most important and salient areas or targets within the current second. Sometimes, drivers may focus on more than one salient area or target. They direct their attention to surrounding objects (such as vehicles, pedestrians, traffic signs, and other visual stimuli) based on specific scene, task-related, and object-level cues.

[0004] Therefore, a driver attention area prediction method is needed to reasonably allocate the driver's attention and correctly process the information of the area of ​​interest to improve the driver's driving safety. Summary of the Invention

[0005] In view of this, the present invention provides a driver attention area prediction system, which can accurately predict the driver's attention distribution in four modes: left-hand drive right-hand drive safe situation, left-hand drive right-hand drive dangerous situation, right-hand drive left-hand drive safe situation and right-hand drive left-hand drive dangerous situation.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for predicting a driver's attention area comprises the following steps:

[0008] S1: obtaining an original driving video of the driver's attention area, and preprocessing the original driving video to obtain a processed driving video sequence;

[0009] S2: Use the deep learning framework Keras to build a deep learning model and train the deep learning model;

[0010] S3: Use the trained deep learning model to perform region prediction on the pre-processed driving video sequence to obtain the driver's attention area prediction result;

[0011] S4: Output the driver's attention area prediction result.

[0012] Furthermore, in step S1, preprocessing the original driving video includes the following steps:

[0013] Preprocessing the original driving video includes the following steps:

[0014] S1.1: Select the mode of original driving video input;

[0015] S1.2: Selecting the original driving video to be predicted based on the mode of the original driving video input;

[0016] S1.3: Based on the selection result, the original driving video to be predicted is preprocessed and prepared into a data set to obtain a processed driving video sequence.

[0017] Furthermore, in step S1.1, the mode of selecting the driving video input includes four driving modes: left-hand drive safe driving, left-hand drive dangerous driving, right-hand drive safe driving, and right-hand drive dangerous driving.

[0018] Furthermore, the preprocessing of the original driving video to be predicted includes preprocessing the acquired original driving video, abnormal video filtering, abnormal video repairing, missing video compensation, duplicate video deletion, image grayscale processing, and image normalization processing.

[0019] Furthermore, in step S2, the steps of constructing a deep learning model training model using the deep learning keras framework and training the deep learning model include:

[0020] S2.1: Color feature extraction: Based on the C3D algorithm and the CBMA serial encoding module, the original input video sequence is processed for RGB color information and bilinear sampling is performed to extract color features.

[0021] S2.2: Shape feature extraction: Use dilated convolution to perform semantic segmentation on the shape, process temporal information based on the C3D encoding module, and extract shape features through bilinear sampling;

[0022] S2.3: Motion feature extraction: Use the dense optical flow algorithm to obtain the optical flow field to extract motion information, and then process the timing information based on the C3D encoding module, and extract motion features through bilinear sampling;

[0023] S2.4: Feature fusion, using a multi-layer deep feature fusion algorithm to fuse color features, shape features, and motion features to obtain the final FOA.

[0024] Furthermore, the outputting of the driver attention area prediction result includes displaying the prediction video result, storing the video in the same directory of the original driving video, and directly jumping to the driving video playback interface where the prediction has been completed.

[0025] Furthermore, a driver attention area prediction system is provided, the system comprising:

[0026] A video acquisition and preprocessing module is used to acquire the original driving video of the driver's attention area and preprocess the original driving video to obtain a processed driving video sequence;

[0027] Model training module, used to build and train deep learning models;

[0028] The model application module uses the trained deep learning model to perform regional prediction on the pre-processed driving video sequence to obtain the driver's attention area prediction results;

[0029] The result display module is used to display the predicted driving video results.

[0030] Furthermore, the video acquisition and preprocessing module includes a video acquisition module and a video preprocessing module. The video acquisition module includes a video mode selection unit and a video selection unit. The video mode selection module is used to select the mode of the original driving video and send the mode result to the video selection unit. The video selection unit is used to select the original driving video that needs to be predicted; the video preprocessing module is used to preprocess the acquired original driving video, filter abnormal videos, repair abnormal videos, compensate for missing videos, and delete duplicate videos.

[0031] Furthermore, the model training module includes a color feature extraction module, a shape feature extraction module, a motion feature extraction module and a fusion module. The color feature extraction module is used to process the RGB color information of the original input video sequence based on the serial encoding module of C3D and CBMA. The shape feature extraction module is used to perform semantic segmentation on the shape using void convolution. The motion feature extraction module is used to obtain the optical flow field using dense optical flow to obtain motion information. The fusion module is used to fuse the color features, shape features and motion features using a multi-layer fusion algorithm to obtain the final FOA.

[0032] It can be seen from the above technical solution that the advantages of the present invention are:

[0033] Compared with the existing technology, the present invention has the ability to perform differentiated training on driving attention in LHT and RHT scenarios, and can accurately predict driving attention by scenario and situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0035] Figure 1 This is a step diagram of a method for predicting a driver's attention area according to the present invention.

[0036] Figure 2 The figure is a schematic diagram of the composition structure of a driver attention area prediction system of the present invention.

[0037] Figure 3 Schematic diagram of the composition structure of the video acquisition and preprocessing module.

[0038] Figure 4 A diagram showing the steps of video acquisition and preprocessing methods.

[0039] Figure 5 Schematic diagram of the composition structure of the deep prediction system.

[0040] Figure 6 This is a schematic diagram of the composition structure of a UI interface module of the present invention.

[0041] Figure 7 This is a schematic diagram of the composition structure of another UI interface module of the present invention.

[0042] Figure 8 This is a front-end operation flow chart of the driving attention area prediction system of the present invention.

[0043] Figure 9Schematic diagram of the composition structure of the fusion module.

[0044] Figure 10 Schematic diagram of the composition structure of the C3D encoding module of the present invention.

[0045] Figure 11 Schematic diagram of the composition structure of CBMA of the present invention.

[0046] Figure 12 This is the driver attention area prediction effect diagram of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0048] like Figure 1 A method for predicting a driver's attention area is shown, and the method comprises the following steps:

[0049] S1: obtaining an original driving video of the driver's attention area, and preprocessing the original driving video to obtain a processed driving video sequence;

[0050] S2: Use the deep learning framework Keras to build a deep learning model and train the deep learning model;

[0051] S3: Use the trained deep learning model to perform region prediction on the pre-processed driving video sequence to obtain the driver's attention area prediction result;

[0052] S4: Output the driver's attention area prediction result.

[0053] like Figure 2 A driver attention area prediction system is shown, which includes a video acquisition and preprocessing module, a model training module, a model application module and a result display module. The video acquisition and preprocessing module, the model training module, the model application module and the result display module are connected in sequence.

[0054] like Figure 3The video acquisition and preprocessing module shown is used to acquire the original driving video of the driver's attention area and preprocess the original driving video to obtain a driving video sequence. The video acquisition and preprocessing module includes a video acquisition module and a video preprocessing module. The video acquisition module includes a video mode selection unit and a video selection unit. The video mode selection module is used to select the mode of the original driving video and send the mode result to the video selection unit, which is used to select the original driving video to be predicted. The video preprocessing module is used to preprocess the acquired original driving video, filter abnormal videos, repair abnormal videos, compensate for missing videos, and delete duplicate videos.

[0055] like Figure 4 The video acquisition and preprocessing method shown includes the following steps:

[0056] S1.1: Select the mode of original driving video input;

[0057] S1.2: Selecting the original driving video to be predicted based on the mode of the original driving video input;

[0058] S1.3: Based on the selection result, the predicted original driving video is preprocessed and prepared into a dataset to obtain a driving video sequence.

[0059] The original driving video modes include four driving modes: left-hand drive safe driving, left-hand drive dangerous driving, right-hand drive safe driving and right-hand drive dangerous driving.

[0060] The preprocessing of the original driving video to be predicted includes preprocessing the acquired original driving video, abnormal video filtering, abnormal video repairing, missing video compensation, duplicate video deletion, image grayscale processing, and image normalization processing.

[0061] In this embodiment, the prediction system is implemented in the Matlab2019b UI interface and on the IDE platform. Figure 5 As shown in the figure, it mainly includes two parts: front-end and back-end. The front-end of the system mainly obtains videos for driving video mode selection, driving video selection and result display. Figure 6-7 As shown, the system displays the selection of driving video modes and driving videos on the UI interface. The selection of driving video modes includes four driving modes: left-hand drive safe driving, left-hand drive dangerous driving, right-hand drive safe driving, and right-hand drive dangerous driving. The driving scene is determined by the driving video mode selection module. The driving video selection includes an "OK" button and a "Return" button. The driving video selection module is used to determine the video file to be predicted. Figure 8The figure below shows the front-end operational flow chart for the Driving Attention Area Prediction System. On the main interface of the Attention Prediction System, select the prediction mode for different scenarios. Select a specific driving mode from the four driving modes: "Left-hand Drive Safe Driving," "Left-hand Drive Dangerous Driving," "Right-hand Drive Safe Driving," or "Right-hand Drive Dangerous Driving." Click the "OK" button to proceed to the "Select Local Video File to Predict" section. Clicking the "OK" button in the "Select Local Video File to Predict" section will proceed to the "Model Inference" section and the "Prediction Video Display." Clicking the "Back" button will return to the "Mode Selection" interface.

[0062] The system backend is mainly built on a deep learning model. The model is constructed using the Keras framework. Color, shape and motion features are used to predict FOA respectively. Finally, the final FOA is obtained by fusing the features to realize the training of the driving video attention model. The system backend uses the encapsulated algorithm model and the video obtained by the video acquisition and preprocessing module to perform model inference to obtain the result prediction of the driving video attention area. The backend model inference function response is called by clicking the button component on the front end of the system, and then the video obtained by the video acquisition and preprocessing module is processed, and the obtained model results are saved.

[0063] like Figure 9 As shown, the steps to build a deep learning model are as follows:

[0064] S2.1: Color feature extraction: Based on the serial encoding module of C3D and CBMA, the RGB color information of the original input driving video sequence is processed and bilinear sampling is performed to extract color features;

[0065] S2.2: Shape feature extraction: Use dilated convolution to perform semantic segmentation on the shape, process temporal information based on the C3D encoding module, and extract shape features through bilinear sampling;

[0066] S2.3: Motion feature extraction: Dense optical flow is used to obtain the optical flow field to obtain motion information. The C3D encoding module processes the timing information and extracts motion features through bilinear sampling.

[0067] S2.4: Feature fusion, using a multi-layer deep feature fusion algorithm to fuse color features, shape features, and motion features to obtain the final FOA.

[0068] The color feature extraction steps are as follows:

[0069] The serial feature fusion method is used to classify and identify the features and obtain the prediction effect of color features. That is, the serial encoding module based on the three-dimensional convolutional neural network C3D and CBMA is used to process the RGB color information of the original driving video, and color features are extracted through bilinear sampling.

[0070] Among them, Figure 10 Figure 1 is a schematic diagram of the network structure of the C3D encoding module, which obtains the temporal information of the driving video sequence through the C3D algorithm.

[0071] like Figure 11 The network structure diagram of CBMA is shown in Figure 1. CBMA is a lightweight universal module that can be seamlessly integrated into any CNN architecture and trained end-to-end with the basic CNN. CBMA enables the model to focus on more useful information. The CBMA process is divided into two independent parts: channel attention and spatial attention. In the channel attention model, the feature F∈R is first input. ×W×C Where H represents the height, W represents the width, and C represents the number of channels in the feature plane. Max pooling and average pooling are used to compress the spatial dimensions, resulting in two 1×1×C channel descriptions. These are then fed into a two-layer neural network. The first layer has C / R neurons and uses the ReLU activation function, while the second layer has C neurons. These two layers of the neural network are shared. The two features are then added together and weighted using a Sigmoid activation function. Finally, the weighted coefficients are multiplied by the original feature F to obtain the scaled new feature F′.

[0072] In the spatial attention model, the processing method is similar to the channel attention model. Given a new feature F′∈R H×W×C Similarly, two H×W×1 channel descriptions are obtained through max pooling and average pooling. These are then added together and passed through a 7×7 convolutional layer with a Sigmoid activation function to obtain the weight coefficients. Finally, the weight coefficients are multiplied by the feature F' to obtain the scaled new feature F″, which is the spatial saliency feature. Because channel attention and spatial attention complement each other, CBMA can comprehensively filter feature space information. Finally, a bilinear upsampling (restoring the representation to the resolution of the input image) and a final two-dimensional convolution are performed to obtain the field of view (FOA) of the color branch.

[0073] Shape feature extraction: The dilated convolutional network used has eight layers. The first seven layers have a 3x3 kernel size, and the last layer has a 1x1 kernel size. Because dilated convolutions increase the image size, edge clipping is performed in the first seven layers. At the same time, the network expands from small to large, shifting from perceiving local features in a small area to distributing features across a larger area through large convolutions. A C3D-based encoding module is used to extract temporal information, ultimately yielding the field of view (FOA) of the shape branch.

[0074] Motion feature extraction: Leveraging the temporal changes in pixels within an image sequence and the correlation between adjacent frames, we identify the correspondence between the previous and current frames and calculate the motion of objects between them. Dense optical flow performs point-by-point matching on the image, calculating the offset of all points to generate the optical flow field. Finally, a C3D-based encoding module extracts temporal information, ultimately yielding the field of view (FOA) of the motion branch.

[0075] Feature Fusion: Utilizing a multi-scale, multi-branch deep feature fusion algorithm and feature learning from convolutional neural networks, the algorithm first learns low-level features such as coolor, shape, and motion. These low-level features are then convolved to a certain dimension through convolution operations and fed back to high-level features. Finally, a feature fusion strategy is used to fuse multiple features. Leveraging the structural information inherent in features of varying depth, the algorithm achieves complementary fusion between low-level and high-level features, as well as between different features, ultimately achieving the predicted results of model training, namely the driver attention area prediction model.

[0076] The model application module uses the trained deep learning model to perform regional prediction on preprocessed driving video sequences, obtaining driver attention area prediction results. The model application module is written in Matlab and encapsulated as a callable function. This function is called through a Matlab logic program to complete the driving attention prediction for the input video and obtain the desired driver attention area prediction.

[0077] The result display module is used to display the predicted video results, store the video in the same directory as the original driving video, and directly jump to the driving video playback interface that has been predicted.

[0078] After completing the model training, the trained model is used for model inference. The trained and encapsulated model is applied to process the driving video sequence obtained by the video acquisition and preprocessing module, and the predicted video results are displayed. The video is stored in the same directory as the original driving video, and the predicted driving video playback interface is directly jumped to. Figure 12 The driver's attention area prediction effect diagram is shown.

[0079] The present invention can predict driving attention in safe and dangerous situations in both LHT and RHT driving scenarios, respectively, to better understand how to correctly allocate driving attention, which has far-reaching significance for novice drivers to learn attention allocation and autonomous driving.

[0080] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for predicting a driver's attention area, characterized in that: The following steps are involved: S1: obtaining an original driving video of the driver's attention area, and preprocessing the original driving video to obtain a processed driving video sequence; S2: Use the deep learning framework Keras to build a deep learning model and train the deep learning model; S3: Use the trained deep learning model to perform region prediction on the pre-processed driving video sequence to obtain the driver's attention area prediction result; S4: Output the prediction result of the driver's attention area; In step S2, the deep learning model training model is constructed using the deep learning keras framework, and the steps of training the deep learning model include: S2.1: Color feature extraction: Based on the C3D algorithm and the CBMA serial encoding module, the original input video sequence is processed for RGB color information and bilinear sampling is performed to extract color features. S2.2: Shape feature extraction: Use dilated convolution to perform semantic segmentation on the shape, process temporal information based on the C3D encoding module, and extract shape features through bilinear sampling; S2.3: Motion feature extraction: Use the dense optical flow algorithm to obtain the optical flow field to extract motion information, and then process the timing information based on the C3D encoding module, and extract motion features through bilinear sampling; S2.4: Feature fusion, using a multi-layer deep feature fusion algorithm to fuse color features, shape features, and motion features to obtain the final FOA.

2. The method for predicting the driver's attention area according to claim 1, characterized in that: In step S1, preprocessing the original driving video includes the following steps: S1.1: Select the mode of original driving video input; S1.2: Selecting the original driving video to be predicted based on the mode of the original driving video input; S1.3: Based on the selection result, the original driving video to be predicted is preprocessed and prepared into a data set to obtain a processed driving video sequence.

3. The method for predicting the driver's attention area according to claim 2, characterized in that: In the step S1.1, the mode of selecting the driving video input includes four driving modes: left-hand drive safe driving, left-hand drive dangerous driving, right-hand drive safe driving, and right-hand drive dangerous driving.

4. The method for predicting the driver's attention area according to claim 2, characterized in that: The preprocessing of the original driving video to be predicted includes preprocessing the acquired original driving video, abnormal video filtering, abnormal video repairing, missing video compensation, duplicate video deletion, image grayscale processing, and image normalization processing.

5. The method for predicting the driver's attention area according to claim 1, characterized in that: The output of the driver attention area prediction result includes displaying the prediction video result, storing the video in the same directory as the original driving video, and directly jumping to the driving video playback interface that has been predicted.

6. A driver attention area prediction system, characterized in that: The system comprises: A video acquisition and preprocessing module is used to acquire the original driving video of the driver's attention area and preprocess the original driving video to obtain a processed driving video sequence; Model training module, used to build and train deep learning models; The model application module uses the trained deep learning model to perform regional prediction on the pre-processed driving video sequence to obtain the driver's attention area prediction results; The result display module is used to display the predicted driving video results; The video acquisition and preprocessing module includes a video acquisition module and a video preprocessing module. The video acquisition module includes a video mode selection unit and a video selection unit. The video mode selection module is used to select the mode of the original driving video and send the mode result to the video selection unit. The video selection unit is used to select the original driving video to be predicted; the video preprocessing module is used to preprocess the acquired original driving video, filter abnormal videos, repair abnormal videos, compensate for missing videos, and delete duplicate videos. The model training module includes a color feature extraction module, a shape feature extraction module, a motion feature extraction module and a fusion module. The color feature extraction module is used to process the RGB color information of the original input video sequence based on the serial encoding module of C3D and CBMA. The shape feature extraction module is used to perform semantic segmentation on the shape using void convolution. The motion feature extraction module is used to obtain the optical flow field using dense optical flow to obtain motion information. The fusion module is used to fuse the color features, shape features and motion features using a multi-layer fusion algorithm to obtain the final FOA.

Images