Driver attention distribution monitoring method based on double monocular cameras

Abstract

The invention discloses a driver attention distribution monitoring method based on double monocular cameras, and relates to the field of automatic driving safety, the method comprises the steps that two monocular cameras are arranged in a vehicle, the first camera monitors the driving behavior and the head posture of a driver, and the second camera is used for shooting the area watched by the driver in the driving process; carrying out region division on the image shot by the camera II, and numbering; using an Openface model to identify the number of the area watched by the driver in the step (3); obtaining a fixation point thermodynamic diagram of the AOI by using a ViT-Attention Rollout model pair; integrating the thermodynamic diagram, the piece data and the vehicle behavior data to carry out attention distribution analysis, and determining attention distribution modes of the driver in different driving environments and influence degrees of different events on attention distribution; according to the invention, related data acquisition is carried out through the two monocular cameras, so that interference caused by intrusive sight capturing equipment is eliminated.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving safety, and in particular to a driver attention allocation monitoring method based on dual monocular cameras. Background Art

[0002] With the rapid development of self-driving cars, more and more entertainment and life service functions unrelated to driving have been incorporated into the human-computer interaction system of vehicles. Many of these functions require drivers to stare at the screen for a long time, which will cause great visual distraction. Traffic driving scenes are complex and diverse. Experienced drivers usually allocate visual attention to key targets and areas in advance and judge their intentions to ensure driving safety. Therefore, the study of driver visual attention allocation behavior is crucial to the development of assisted driving systems and autonomous driving.

[0003] At present, most monitoring devices used to evaluate the driver's attention allocation on driving tasks use eye trackers. The eye tracker's line of sight capture effect is very accurate, which can provide more reliable data and compare the driver's actual scanning path with the designer's ideas. But it also has some limitations. For example: the eye tracker can measure the position where the eyes are focused, but it cannot monitor the driver's psychological state and behavioral intentions; secondly, the complexity and variability of the driving environment may affect the detection effect of the eye tracker; most importantly, the invasive line of sight capture device will affect or even change the psychology and behavior of the monitored object, and it is difficult to conduct experiments on people with visual impairments who need to wear glasses. This influence will lead to a lack of universality in the sample data and may also distort the experimental results.

[0004] Based on this, the present invention aims to propose a driver attention allocation monitoring method based on dual monocular cameras to eliminate the interference caused by intrusive sight capture devices. Summary of the invention

[0005] In order to solve the impact of intrusive gaze capture devices on the evaluation of driver attention allocation, the present invention provides a driver attention allocation monitoring method based on dual monocular cameras, which eliminates the interference caused by intrusive gaze capture devices by collecting relevant data through two monocular cameras.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A driver attention allocation monitoring method based on dual monocular cameras comprises the following steps:

[0008] (1) Two monocular cameras are installed in the car, where camera 1 monitors the driver's driving behavior and head posture, and camera 2 is used to capture the area the driver is looking at while driving;

[0009] (2) Divide the image captured by camera 2 into regions and number them;

[0010] (3) Training the Openface model to identify the driver's gaze area based on head posture and eye gaze; and using the trained Openface model to perform facial recognition on the image captured by camera 1 to identify the area number in step (3) that the driver is looking at;

[0011] (4) The ViT-AttentionRollout model is used to process the numbered area where the driver is looking and obtain the gaze point heat map of the AOI;

[0012] (5) Obtain road traffic events using images captured by camera 2, obtain vehicle behavior based on vehicle sensors, and conduct attention allocation analysis based on the thermal map, event data, and vehicle behavior data to determine the driver's attention allocation pattern in different driving environments and the degree of influence of different events on attention allocation.

[0013] In the present invention, the image captured by the second camera is divided into the front windshield, the left rearview mirror, the right rearview mirror, the middle console, the central rearview mirror, and the speedometer area.

[0014] In the present invention, images of the driver's facial features during driving are obtained, the driver's gaze areas in these images are manually classified, and the images are used as a data set for training the Openface model.

[0015] In the present invention, the road traffic events are obtained using the Yolov8 algorithm, and the Yolov8 algorithm is used to collect the road environment, traffic signals, vehicle behavior, chronic traffic behavior, and road signs in the image taken by the second camera.

[0016] In the present invention, vehicle behaviors include acceleration, deceleration, braking, and steering.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. The driver attention allocation monitoring method of the present invention is realized by two monocular cameras. Camera 1 is used to shoot driving behavior and head posture. The Openface model is used for analysis to determine the area where the driver's gaze is fixed. Camera 2 is used to shoot the image that the driver can see during driving. The gaze is projected onto the image shot by camera 2 to obtain the driver's gaze area. The ViT-AttentionRollout model is used to analyze the area where the driver is fixed to obtain the driver's gaze point. The present invention associates the two images to more accurately analyze the driver's gaze point, which is beneficial for subsequent attention allocation analysis.

[0019] 2. In the present invention, the driver's attention distribution in a natural driving environment can be obtained through the ViT-Attention Rollout model. In the ViT model, convolution is used to segment image blocks, improve the underlying feature extraction performance of the model, and improve the target recognition accuracy. In terms of model selection, the ViT model reduces the number of parameters in the model, reduces the computational complexity, introduces a multi-head attention mechanism, can effectively capture the global information in the image, improves the performance in large-scale data sets, and can process images in real time, and can show better results on small sample data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of the installation position of Camera 1 of the present invention.

[0021] Figure 2 Schematic diagram of facial landmarks in the present invention.

[0022] Figure 3 Schematic diagram of the ViT model structure in the present invention.

[0023] Figure 4 This is the structural diagram of the multi-head attention mechanism in the present invention.

[0024] Figure 5 This is the structural diagram of Embedding attentions in the present invention.

[0025] Figure 6 This is the Attention Rollout structure diagram in the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] A driver attention allocation monitoring method based on dual monocular cameras, comprising:

[0028] Step 1: Image acquisition

[0029] Two monocular cameras are installed in the car, where camera one monitors the driver's driving behavior and head posture, and camera two is used to capture the area the driver is looking at while driving;

[0030] Camera 1 is placed at the same position as the central rearview mirror to monitor the driver's driving behavior and head posture. Figure 1 As shown; Camera 2 is placed above the driver's head to obtain the human-computer interaction situation of the entire area of ​​interest (AOI) under the vision of Camera 2.

[0031] Step 2: AOI area image division

[0032] The image captured by the second camera is divided into regions, specifically into the front windshield, the left rearview mirror, the right rearview mirror, the middle console, the central rearview mirror, and the speedometer region, and each region is numbered;

[0033] (3) Gaze area recognition

[0034] The Openface model is trained to identify the driver's gaze area based on the head posture and eye gaze; and the trained Openface model is used to perform facial recognition on the image captured by camera 1 to identify the area number in step (3) where the driver is looking;

[0035] Facial and head features are extracted through the Openface model, which can perform facial landmark detection and action unit recognition, head posture and eye gaze estimation, and allocate the driver's gaze to different areas to provide information about the driver's attention.

[0036] The Openface model starts with face detection, followed by 68 facial landmark detections, such as Figure 2 As shown. These landmarks are used to estimate head pose and track eye gaze, and the constrained local neural field model (CLNF) framework is used as a shape registration method to detect facial features. The CLNF algorithm mainly consists of two parts: point distribution model (PDM) and patch model. CLNF combines the patch model of the local neural field (LNF) and uses an uneven regularized feature point mean shift method for face fitting during the fitting process. PDM captures changes in the shape of the markers, and the local appearance changes of each marker are captured by the patch model.

[0037] (4) The ViT-AttentionRollout model is used to process the area where the driver is looking and obtain the gaze point heat map of the AOI;

[0038] The ViT-AttentionRollout model processes the area the driver is looking at as follows:

[0039] (4.1) AOI area image block segmentation

[0040] Here, 2D convolution is used for image segmentation, dividing the input image into multiple blocks of equal size, and each block is regarded as a sequence. Perform 2D convolution, where H i-1 ×W i-1 is the resolution of the original image, C i-1 is the number of channels, which is usually 3. If the kernel size of the 2D convolution is k×k and the step size is p, the output height and width are:

[0041]

[0042] The feature dimension after 2D convolution becomes H i ×W i ×C', where Let M i =H i ×W i Then the three-dimensional features are flattened Convert to 2D features

[0043] Transformer uses a fixed input vector size D in all its layers, so each segmented small image block (patch) needs to be stretched and tiled and mapped to D dimensions using linear projection. The output is called Patch Embedding (PE).

[0044] (4.2) Embedded location information

[0045] Then, position encoding is added to each block so that the model can capture the relative position information between blocks, and the stretched data is used as the position encoding information, such as Figure 4 As shown. Embed location information in PE:

[0046]

[0047] Before embedding the position information, the state at the output of the Transformer encoder is represented as:

[0048]

[0049] During model pre-training and fine-tuning, such a classification head is attached to The classification head is implemented by an MLP hidden layer during pre-training and a linear layer during fine-tuning:

[0050] z′ l =MSA(LN(z l-1 ))+z l-1 , l=1...L

[0051] z l =MLP(LN(z′ l ))+z′l , l=1...L

[0052] (4.3) Transformer Encoder

[0053] like Figure 5 As shown in Figure 1, the Transformer encoder consists of alternating multi-head self-attention layers and multi-layer perceptron blocks (MLP), with layer normalization applied before each module. In the Transformer block, both the self-attention and feed-forward networks are wrapped with residuals, and the inputs of these modules are added to the output.

[0054] In the attention module, the input Transformed into three parts through linear transformation, namely query value Key-value and value Among them, M i +1 is the sequence length, C′, C k , C v are the input, query value (key value) and value dimensions respectively, and attention is expressed as:

[0055]

[0056] Multi-head self-attention splits the query, key, and value into h parts, where h is the number of heads in the multi-head attention module. The output value of each head is concatenated and linearly projected to form the final output:

[0057]

[0058] MultiHead(Q,K,V)=Concat(head1,...,head h )

[0059] MLP is used for nonlinear feature transformation, and this process can be expressed as:

[0060] MLP(x)=FC(σ(FC(x)))

[0061] FC(x)=xW+b

[0062] Where: FC represents the fully connected layer; W and b are the weight and bias of the fully connected layer respectively; σ(·) represents the activation function.

[0063] Layer normalization is a key component of the Transformer structure and is expressed as

[0064]

[0065] Where: LN represents layer normalization; x is the input; μ and δ are the mean and standard deviation of the features respectively; represents the multiplication of each element of a vector or matrix; γ and β are learnable affine transformation parameters.

[0066] For input The Transformer encoder structure can be expressed as:

[0067]

[0068] Let the process of Transformer encoder be TE(x), and the process of extracting category Token can be expressed as:

[0069]

[0070] Among them, CT represents a single Token. In ViT, a Token does not represent a pixel, but a complete patch block; TE represents the Transformer encoder.

[0071] (4.4) Attention Rollout

[0072] AttentionRollout takes the embedding attention as input and recursively calculates the token attention value in each layer of a given model.

[0073] In order to calculate how information propagates from the input layer to the embedding of higher layers, it is crucial to consider the residual connections in the model as well as the values ​​of the attention weights. In the Transformer block, both the self-attention and feed-forward networks are wrapped with residual connections, and the inputs of these modules are added to the output. In the model, only the attention weights are used to approximate the information flow in the Transformer, ignoring the residual connections, but these connections play an important role in bundling corresponding positions in different layers. Therefore, in order to calculate the AttentionRollout, the attention map needs to be enhanced with additional weights to represent the remaining connections. Given an attention module with residual connections, the value of the l+1 layer is calculated as V l+1 =V l +W att V l , where W att is the attention matrix, so we have V l+1 =(W att +I)V l To further illustrate the residual connection, add a unit matrix to the attention matrix and renormalize the weights. We get A = 0.5W att +0.5I, where A is the original attention updated by the residual connection.

[0074] In addition, analyzing individual heads requires analyzing the mixed information between heads through the position-wise feed-forward network in the Transformer block. Each head can also be analyzed individually using Attention Rollout. Attention Rollout is an intuitive method that can trace the information propagated from the input layer to the higher-layer embeddings. Given a Transformer with L layers, calculate the attention from all positions in layer l i to layer l j where j < i. As Figure 6 shown, in the attention map, the path from the node v at position k in l i to the node u at position m in l j is a series of routes connecting these two nodes. If the weight of each edge is regarded as the proportion of information transmitted between two nodes, the amount of information at v propagated to u through a specific path can be calculated by multiplying the weights of all edges in this path. Since there may be more than one path between two nodes in the attention map, to calculate the total amount of information propagated from v to u, sum over all possible paths between these two nodes. At the implementation level, to calculate the attention from l i to l j , recursively multiply the attention weight matrices of all layers:

[0075]

[0076] where A~ is Attention Rollout, A is the original attention, and for convenience in calculating attention, let j = 0. Finally, obtain the fixation heatmap.

[0077] (5) Obtain road traffic events using the images captured by Camera 2, acquire vehicle behaviors based on vehicle sensors, and conduct attention allocation analysis by integrating the heatmap and event data and vehicle behavior data to determine the driver's attention allocation patterns in different driving environments and the influence degree of different events on attention allocation.

[0078] First, obtain video data in the natural driving environment through Camera 2, use the Yolov8 algorithm to perform object detection based on the perspective of Camera 2, and collect road traffic event data in the driving environment, including traffic signals, vehicle behaviors, road signs, sudden events during driving, etc. information. At the same time, obtain the available parameters in the environment and the vehicle's own parameters affected by the driver's actions such as stepping on the pedal and steering, and the changes in vehicle actions are transmitted through sensors;

[0079] Subsequently, perform time synchronization operations on the fixation heatmap and road traffic events to ensure the accuracy and consistency of the fixation information and the occurrence time of road traffic events;

[0080] Secondly, analyze the road traffic event data and identify different events that occurred in the video recording, such as vehicle lane changes, traffic signal changes, pedestrian crossings, etc., which can be achieved by calculating the event, location and type of the event; further, combine the heat map and road traffic event data, by superimposing the event data on the heat map, and analyzing the distribution of gaze points when the event occurs. For example, it can be determined whether the driver noticed the event when it occurred: when the target detection algorithm recognizes the traffic light and the left (right) turn signal of the vehicle is on, detect whether the driver looks at the left (right) rearview mirror area; whether the driver looks at the middle console for human-computer interaction when the vehicle is stationary; when the vehicle is running normally and the speed remains unchanged, whether the driver's gaze is always on the front windshield;

[0081] Finally, based on the analysis results combining the heat map and event data, attention allocation analysis is performed, which includes determining the driver's attention allocation pattern in different driving environments and the degree of influence of different events on attention allocation.

[0082] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A driver attention allocation monitoring method based on dual monocular cameras, characterized in that: The following steps are involved: (1) Two monocular cameras are installed in the car, where camera 1 monitors the driver's driving behavior and head posture, and camera 2 is used to capture the area the driver is looking at while driving; (2) Divide the image captured by camera 2 into regions and number them; (3) Training the Openface model to identify the driver's gaze area based on head posture and eye gaze; and using the trained Openface model to perform facial recognition on the image captured by camera 1 to identify the area number in step (3) that the driver is looking at; (4) The ViT-AttentionRollout model is used to process the numbered area where the driver is looking and obtain the gaze point heat map of the AOI; (5) Obtain road traffic events using images captured by camera 2, obtain vehicle behavior based on vehicle sensors, and conduct attention allocation analysis based on the thermal map, event data, and vehicle behavior data to determine the driver's attention allocation pattern in different driving environments and the degree of influence of different events on attention allocation.

2. The driver attention allocation monitoring method based on dual monocular cameras according to claim 1 is characterized in that: The image captured by camera 2 is divided into the front windshield, left rearview mirror, right rearview mirror, center console, central rearview mirror, and speedometer areas.

3. The driver attention allocation monitoring method based on dual monocular cameras according to claim 2 is characterized in that: Images of the driver's facial features during driving are obtained, the driver's gaze areas in these images are manually classified, and these images are used as a dataset to train the Openface model.

4. The driver attention allocation monitoring method based on dual monocular cameras according to claim 1 is characterized in that: The road traffic events are obtained using the Yolov8 algorithm, which collects the road environment, traffic signals, vehicle behaviors, chronic traffic behaviors, and road signs in the images taken by camera two.

5. The driver attention allocation monitoring method based on dual monocular cameras according to claim 1 is characterized in that: Vehicle behaviors include acceleration, deceleration, braking, and steering.

Images