Driver distraction detection method based on gaze direction contrast learning

Abstract

The present invention discloses a distracted driving behavior detection method based on gaze direction contrast learning. The present invention designs a convolutional neural network to estimate the gaze point coordinates of the driver's head, and calculates the gaze direction according to the gaze point coordinates. The position of the target of interest in the driving scene is considered to find the target of interest in the gaze direction, and the gaze direction is corrected. The present invention accurately identifies the easily confused driving state by mining the easily confused driving state video frames and using contrast learning. Contrastive learning considers that the easily confused safety video frame features are close to the easily recognizable safety video frame features, and at the same time, they are far away from the easily recognizable distraction video frame features, thereby effectively separating the easily confused distraction video frames and safety video frames. The gaze direction estimated by the present invention points to a clear target of interest. At the same time, the use of contrast learning can effectively realize distraction behavior detection in the case of easily confused gaze directions.

Description

Technical Field

[0001] The present invention relates to the technical field of person sight direction detection, and in particular to a driver distraction behavior detection method based on sight direction comparative learning. Background Art

[0002] In recent years, with the increasing number of private cars, traffic accidents have also increased. Statistics show that distracted driving, such as using a mobile phone, holding something, or looking away for extended periods of time, is a major cause of road accidents. Therefore, developing a high-performance distracted driving detection system that monitors drivers in real time and provides appropriate warnings can significantly reduce traffic accidents caused by distracted driving.

[0003] Chinese patent application publication number CN114049661A, "A distracted driving detection method based on the combination of facial keypoint recognition and support vector machines," proposes a distracted driving detection method based on the combination of facial keypoint recognition and support vector machines. This method first locates facial features in video frames using a keypoint recognition model. Then, a support vector machine is used to model the keypoint positional features of each driver's facial video to classify the driver's distracted driving status. Chinese patent application publication number CN114241453A, "A driver distracted driving monitoring method using keypoint attention," proposes a driver distracted driving monitoring method using keypoint attention. This method uses channel-space transform convolution to improve the feature transformation process, enhancing the representation capability of the convolutional features. Furthermore, it uses projection of keypoint information onto global features to improve the discriminability of the final classification representation, further enhancing the accuracy of driver distracted driving monitoring. Chinese patent application publication number CN114187581A "A fine-grained driver distraction detection method based on unsupervised learning" proposes a fine-grained driver distraction detection method based on unsupervised learning, which makes up for the poor global perception ability of the CNN model stacked by traditional convolutional layers. It has strong local perception ability and good global perception performance, which can enhance the generalization ability of the unsupervised learning detection model for application scenarios.

[0004] In order to make full use of unlabeled datasets, minimize the distance between similar samples, and widen the gap between positive and negative samples, contrastive learning has gradually been used for human action recognition. Yonglong Tian et al. proposed a contrastive learning framework in "Contrastive multiview coding" to maximize the mutual information between different views of the same scene to achieve view-invariant representations. Ting Chen et al. selected negative samples by using enhanced views of other projects in a mini-batch in "A simple framework for contrastive learning of visual representations". Kaiming He et al. used a dynamically updated memory bank of negative representations in "Momentum contrast for unsupervised visual representation learning" to get rid of the batch size limit and make the use of negative samples consistent.

[0005] However, in the process of detecting driver distraction, the position of the driver's eyes and head, as well as the relationship between other objects in the car and the driver, will affect the detection of driver distraction. The above methods do not fully consider the position of the target of interest in the driving scene. In addition, the above methods usually process a single segment separately, ignoring the rich temporal relationships, resulting in difficulty in classifying segments with ambiguous behaviors and backgrounds. Summary of the Invention

[0006] The purpose of the present invention is to make up for the shortcomings of the existing technology and provide a driver distraction behavior detection method based on gaze direction contrast learning. The present invention estimates the driver's gaze direction through the driver's eye features and head position features, and at the same time establishes a graph convolutional network to fully learn the targets of interest in the driver's gaze direction. By mining the driving state video frames that are easily confused, the present invention uses contrast learning to accurately identify the easily confused driving states.

[0007] The present invention is achieved through the following technical solutions:

[0008] A method for detecting driver distraction based on gaze direction contrast learning includes the following steps:

[0009] S1: Process the driver's head video to obtain video frames;

[0010] S2: Build a gaze point extraction network to extract the driver's gaze point coordinates;

[0011] S3: Construct a spatial graph convolutional network weighted by the driver’s sight line and the distance to the object to extract the spatial graph convolution features of the driver’s sight line direction;

[0012] S4: Spatial Graph Convolutional Network Training Based on Contrastive Learning;

[0013] S5: Driver distraction detection based on contrastive learning;

[0014] The specific steps of processing the driver's head video to obtain a video frame in step S1 are as follows:

[0015] S1-1: During driving, use the infrared camera located in front of the driver to obtain the driver's head video V n , where n = 1, 2...N, represents the number of videos and their true value labels 0 means non-distracted driving, 1 indicates distracted driving;

[0016] S1-2: Each video V obtained in step S1-1 n Divide into multiple non-overlapping snippets n,m , where m = 1, 2...M, represents the number of fragments;

[0017] S1-3: For each snippet n,m Perform random sampling to get a video frame n；m ;

[0018] The gaze point extraction network is constructed in step S2 to extract the coordinates of the driver's gaze point. The specific steps are as follows:

[0019] S2-1: The video frame obtained in step 1-3 is n；m Through the head position extraction model, the head position coordinate p is obtained h (The original symbol is Subscripts are omitted for subsequent symbols);

[0020] S2-2: Calculate the loss function of the head position coordinates;

[0021]

[0022] where p h_gt represents the true value of the head coordinate, d(p h ,p h_gt ) indicates p h and p h_gt The Euclidean distance between

[0023] S2-3: Extract the driver’s binocular features;

[0024] S2-3-1: Crop the video frame obtained in S1-2, extract the left and right eye areas, and resize them to 36×60;

[0025] S2-3-2: The normalized left eye area and right eye area are passed through the parallel ResNet-18 network respectively, and the obtained left eye feature is represented as f l , the right eye feature is represented as f r , whose dimensions are all 1×224;

[0026] S2-3-3: Set the left eye feature f l and right eye feature f r The two layers are connected in series through a fully connected layer (FC256) with an output dimension of 256 to obtain the driver’s binocular features f e ;

[0027] S2-4: Predict gaze point coordinates p g ;

[0028] S2-4-1: The head position coordinates p obtained in step S2-1 are h , through three fully connected layers with an output dimension of 256, the head position feature f is obtained h ;

[0029] S2-4-2: The binocular feature f obtained in step S2-3-3 is e and the head position feature f obtained in step S2-4-1 h Connected in series, through two fully connected layers with output dimensions of 256 and 2 respectively, the gaze point coordinate p is obtained g ;

[0030] S2-5: Calculate the loss function of the gaze point coordinates;

[0031] S2-5-1: Calculate the normalized vector from the driver's head to the gaze point:

[0032] S2-5-2: Calculate the loss function of the gaze point coordinates:

[0033]

[0034] where p g represents the predicted gaze point coordinates, p g_gt represents the true value of the gaze point coordinate, p h Represents the predicted head position coordinates;

[0035] Step S3 constructs a spatial graph convolution network weighted by the driver's sight line and the distance to the object to extract the spatial graph convolution features of the driver's sight line direction. The specific steps are as follows:

[0036] S3-1: The video frame image obtained in step S1-3 is passed through the RCNN network to detect the object position o and its feature f in the image o , using it as a node of the spatial graph convolutional network;

[0037] S3-2: Calculate the weighted line of sight between the driver and the object and use it as the edge of the graph convolutional network;

[0038] S3-2-1: Obtain the head position coordinates p of step S2-1 h , the gaze point coordinate p in step S2-4 g and the position o of the object detected in step S3-1, calculate the driver's direction and the position o of the object j j Angles between directions:

[0039]

[0040] S3-2-2: Calculate the weighted relationship between the driver and the object's sight line:

[0041]

[0042] Among them j represents the position of object j, represents Gaussian distribution, σ g represents the parameters of the gaze-weighted Gaussian distribution,

[0043] S3-3: Calculate the distance weighted relationship between the driver and the object and use it as an edge:

[0044]

[0045] Among them i and o j denote the positions of object i and object j respectively, d(·) denotes the Euclidean distance between object i and object j, represents Gaussian distribution, σ d Represents the parameters of the distance-weighted Gaussian distribution;

[0046] S3-4: Calculate the spatial graph convolution features of the driver’s sight direction;

[0047] S3-4-1: Obtain the object feature f detected by RCNN from step S3-1 o ;

[0048] S3-4-2: Obtain the sight-line weighted relationship in step S3-2 and the distance weighted relationship in step S3-3;

[0049] S3-4-3: Calculate the sight-weighted adjacent matrix:

[0050] A g ={α g (o i ,o j )} i,j ,

[0051] where α g (o i ,o j ) represents object o i With object o j The weighted relationship of sight lines between

[0052] S3-4-4: Calculate the sight-weighted spatial graph convolution features:

[0053]

[0054] Among them, A g represents the gaze-weighted adjacency matrix, W g represents the parameters of graph convolution, The dimension is 1×4096;

[0055] S3-4-5: Calculate the distance-weighted adjacency matrix:

[0056] A d ={α d (o i ,o j )} i,j ,

[0057] where α d (o i ,o j ) represents object o i With object o j The distance-weighted relationship between

[0058] S3-4-6: Calculate distance-weighted spatial graph convolution features:

[0059]

[0060] Among them A d Represents the distance-weighted proximity matrix, W d represents the parameters of graph convolution, The dimension is 1×4096;

[0061] S3-4-7: Calculate the spatial graph convolution feature f of the driver's line of sight fusion :

[0062]

[0063] in represents the serial operation, φ fcRepresents a fully connected layer with an output dimension of 1024;

[0064] The spatial graph convolutional network training based on contrastive learning described in step S4 mines difficult and simple segments and calculates the segment contrastive learning loss function. The specific steps are as follows:

[0065] S4-1: Calculate driver behavior score;

[0066] S4-1-1: Constructing the classifier φ cls , which contains a temporal convolution, ReLU activation function and Dropout layer;

[0067] S4-1-2: Obtain the spatial graph convolution feature f of the driver's sight direction through step S3-4 fusion ;

[0068] S4-1-3: Obtain segment-level behavior prediction scores:

[0069] s n,m =φ cls (f fusion ),

[0070] where s n；m represents the behavior prediction score of the mth segment in the nth video, n = 1, 2...N, m = 1, 2...M;

[0071] S4-2: Difficult background segment mining;

[0072] S4-2-1: Add a threshold to the action classification result score obtained in step S4-1-4, and use a step function to generate a binary sequence, where 1 corresponds to action and 0 corresponds to background:

[0073]

[0074] Where ε(·) represents the Heaviside step function, b is 0.5, represents the threshold, and when s n；m >b, If it is 1, it indicates an action clip, otherwise it is 0, indicating a background clip;

[0075] S4-2-2: Apply binary dilation operation (·;*) to the behavior prediction score + , expand the time range of the behavior interval and express the expansion operation as and Where MASK is a large mask and mask is a small mask;

[0076] S4-2-3: Computationally difficult background clip set:

[0077]

[0078] represents the set of difficult background fragments, that is, When it is 0;

[0079] S4-2-4: From the difficult background fragment set Select difficult background clips

[0080]

[0081] in, yes The set of K fragment features in , represents the features of the mth segment in the nth video, yes subset of , where K is a hyperparameter, if If the number of mid-segments is less than K, the reset sampling method is used;

[0082] S4-2-5: Apply binary erosion operation (·;*) to the behavior prediction score - , narrowing the time range of the behavior interval and expressing the erosion operation as and Among them, MASK is a large mask, and mask is a small mask;

[0083] S4-2-6: Collection of fragments of computationally difficult behaviors:

[0084]

[0085] in The set of segments representing difficult behaviors, i.e. When it is 1;

[0086] S4-2-7: From the difficult behavior episode collection Difficult behavior feature set

[0087]

[0088] in, yes The set of K fragment features in , yes subset of , where K is a hyperparameter, if If the number of mid-segments is less than K, the reset sampling method is used;

[0089] S4-3: Simple fragment mining;

[0090] S4-3-1: Score the segment-level behavior prediction score s in step S4-1-3 n；mThe index is sorted in descending order, and the index in step S4-2-3 is removed during the sorting process. and step S4-2-5 The corresponding fragments get the descending index

[0091] S4-3-2: by descending index Selecting a simple behavioral feature set

[0092]

[0093] Where K is a hyperparameter, represent The first K indices of ;

[0094] S4-3-3: Score the segment-level behavior prediction score s in step S4-1-4 n；m The index is sorted in ascending order, and the index in step S4-2-3 is removed during the sorting process. and step S4-2-5 The corresponding fragments get the ascending index

[0095] S4-3-4: indexed in ascending order Selecting a simple background feature set

[0096]

[0097] Where K is a hyperparameter, represent The first K indices of ;

[0098] S4-4: Calculate behavioral loss function;

[0099] S4-4-1: Execute step S4-1-3 to obtain the behavior prediction score s of the segment n,m ;

[0100] S4-4-2: Take s n,m The top k sequences with the highest scores in , and calculate their average value as a n ;

[0101] S4-4-3; the average value a n Through a Softmax function, we get the video-level behavior prediction s n ;

[0102] S4-4-4: Obtaining the true value labels of driver training video categories Calculate the behavior loss function:

[0103]

[0104] S4-5: Calculate the segment contrast loss function;

[0105] S4-5-1: Execute step S4-2-7 to obtain the difficult behavior feature set for Each behavior feature f in step S4-3-2 is randomly selected from the simple behavior feature set Select a behavior feature f + , simple background feature set Select K background features f from - , calculate the loss function:

[0106]

[0107] where f T represents the transposed matrix of f, τ is a hyperparameter, where represents the kth background feature, k = 1, 2...K;

[0108] S4-5-2: Execute step S4-2-4 to obtain the difficult background feature set for Each behavioral feature f in step S4-3-2 is randomly selected from the simple background feature set Select a background feature f from + , simple behavioral feature set Select K background features f from - , calculate the loss function:

[0109]

[0110] where f T represents the transposed matrix of f, τ is a hyperparameter, where represents the kth background feature, k = 1, 2...K;

[0111] S4-5-3: Calculate the segment contrast loss function

[0112]

[0113] in Indicates that Each behavioral feature f in the set of simple behavioral features is randomly selected from Select a behavior feature f + , simple background feature set Select K background features f from - , calculate the mean expected value operation of the loss function; Indicates that Each behavioral feature f in the Select a background feature f + , simple behavioral feature set Select K background features f from - , calculate the mean expected value operation of the loss function;

[0114] S4-6: Calculate the total loss function L of the contrastive learning network for all training videos total :

[0115]

[0116] The balance factor λ is 0.01;

[0117] S4-7: Use the total loss function calculated in step S4-6 as the gradient, and use the back-propagation algorithm to solve the parameters of the model in turn. During the back-propagation process, the parameters that need to be solved include: the parameters of the three fully connected layers in step S2-4-1, the parameters of the two fully connected layers in step S2-4-2, the parameters of the graph convolution in steps S3-4-4 and S3-4-6, and the classifier parameters in step S4-1-3; when the training is completed, the optimal parameter set of the model is obtained.

[0118] The specific steps of the driver distraction behavior recognition based on contrastive learning in step S5 are as follows:

[0119] S5-1: Execute step S1 to obtain a test video of the driver during driving, and perform segmentation and sampling processing to obtain video frames;

[0120] S5-2: Execute step S2 on the obtained video frame to extract the driver's gaze point coordinates p g ;

[0121] S5-3: Execute step S3 to obtain the spatial graph convolution feature f of the driver's sight direction fusion ;

[0122] S5-4: Convolutional feature f is obtained from the spatial graph fusion Use step S4-1-3 to get the segment-level behavior prediction score s n；m ;

[0123] S5-5: The obtained segment-level behavior prediction score s n；m Use step S4-4-3 to get the video-level behavior prediction score s n ,

[0124] S5-6: If the predicted score is s nIf the value is greater than the threshold of 0.6, it indicates distracted driving; otherwise, it does not indicate distracted driving.

[0125] The advantages of the present invention are: the present invention realizes the detection of driver distraction behavior by acquiring the video of the driver during the driving process; to address the problem of inaccurate line of sight direction estimation, the position of the target of interest in the driving scene is considered to find the target of interest in the line of sight direction, and the line of sight direction is corrected to construct a dual-branch graph convolution network based on the target of interest; the dual-branch graph convolution relationship matrix is ​​estimated using the line of sight direction and the observation distance of the target of interest respectively; by jointly considering the direction graph convolution relationship matrix and the distance graph convolution relationship matrix, the target of interest in the scene can be accurately discovered.

[0126] The present invention also considers situations where the proximity of objects of interest can lead to inaccurate gaze direction estimation, resulting in confusion between distracted and safe driving behaviors. To address this issue, the present invention mines video frames of easily confused driving states and utilizes contrastive learning to accurately identify these easily confused driving states. A classifier is used to estimate the time series of distracted driving state scores from the time series of gaze direction estimates generated by a graph convolutional network. Because the scores reflect easily confused distracted and safe driving states, the present invention uses a temporal interval mask to identify easily confused driving state video frames within the temporal context of the confusion point in the time frames surrounding the score confusion point. These frames are further subdivided into easily confused distracted driving behavior video frames and easily confused safe driving behavior video frames based on the scores. Furthermore, outside the temporal range above and below the confusion point, the highest K scores are used to identify easily recognizable distracted driving behavior video frames, while the lowest K scores are used to identify easily recognizable safe driving behavior video frames. Based on this video frame mining, contrastive learning loss optimizes video frame features, aligning the features of easily confused distracted video frames with those of easily recognizable distracted video frames and away from those of easily recognizable safe video frames. Contrastive learning takes into account the characteristics of easily confused safe video frames, which are closer to those of easily recognizable safe video frames, while moving away from those of easily recognizable distracting video frames, thereby effectively separating easily confused distracting video frames from safe video frames. The estimated gaze direction of this invention points to a clear target of interest. Furthermore, using contrastive learning, it can effectively detect distracting behavior even in situations with easily confused gaze directions. BRIEF DESCRIPTION OF THE DRAWINGS

[0127] Figure 1 This is a flow chart of the distracted driving behavior detection method based on gaze direction contrast learning;

[0128] Figure 2 Flowchart for extracting gaze point coordinates from video frames;

[0129] Figure 3 Extracting network graphs for gaze points;

[0130] Figure 4 Spatial graph convolutional network graph weighting the driver's line of sight and distance to objects;

[0131] Figure 5 The binary expansion and erosion operation diagram for the driver behavior prediction score ( Figure 5 (a) is the binary expansion operation diagram for the driver's prediction score; Figure 5 (b) is a binary erosion operation diagram for the driver's predicted score);

[0132] Figure 6 A module for calculating the fragment contrast loss function. DETAILED DESCRIPTION

[0133] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. The present invention is a distracted driving behavior detection method based on sight direction contrast learning. The specific process is as follows Figure 1 As shown, the implementation scheme of the present invention is divided into the following steps:

[0134] S1: Process the driver's head video to obtain video frames;

[0135] S1-1: During driving, use the infrared camera located in front of the driver to obtain the driver's head video V n , where n = 1, 2...N, represents the number of videos and their true value labels 0 means non-distracted driving, 1 indicates distracted driving;

[0136] S1-2: Each video V obtained in step S1-1 n Divide into multiple non-overlapping snippets n,m , where m = 1, 2...M, represents the number of fragments;

[0137] S1-3: For each snippet n,m Perform random sampling to get a video frame n；m ;

[0138] S2: Construct a gaze point extraction network to extract the driver's gaze point coordinates, such as Figure 2 As shown;

[0139] S2-1: The video frame obtained in step 1-3 is n；m Through the head position extraction model, the head position coordinate p is obtained h (The original symbol is Subscripts are omitted for subsequent symbols);

[0140] S2-2: Calculate the loss function of the head position coordinates;

[0141]

[0142] where p h_gt represents the true value of the head coordinate, d(p h ,p h_gt ) indicates p h and p h_gt The Euclidean distance between

[0143] S2-3: Extract the driver’s binocular features;

[0144] S2-3-1: Crop the video frame obtained in S1-2, extract the left and right eye areas, and resize them to 36×60;

[0145] S2-3-2: The normalized left eye area and right eye area are passed through the parallel ResNet-18 network respectively, and the obtained left eye feature is represented as f l , the right eye feature is represented as f r , whose dimensions are all 1×224;

[0146] S2-3-3: Set the left eye feature f l and right eye feature f r The two layers are connected in series through a fully connected layer (FC256) with an output dimension of 256 to obtain the driver’s binocular features f e ;

[0147] S2-4: Predict gaze point coordinates p g ;

[0148] S2-4-1: The head position coordinates p obtained in step S2-1 are h , through three fully connected layers with an output dimension of 256, the head position feature f is obtained h ;

[0149] S2-4-2: The binocular feature f obtained in step S2-3-3 is e and the head position feature f obtained in step S2-4-1 h Connected in series, through two fully connected layers with output dimensions of 256 and 2 respectively, the gaze point coordinate p is obtained g ,like Figure 3 As shown;

[0150] S2-5: Calculate the loss function of the gaze point coordinates;

[0151] S2-5-1: Calculate the normalized vector from the driver's head to the gaze point:

[0152] S2-5-2: Calculate the loss function of the gaze point coordinates:

[0153]

[0154] where p g represents the predicted gaze point coordinates, p g_gt represents the true value of the gaze point coordinate, p h Represents the predicted head position coordinates;

[0155] S3: Construct a spatial graph convolution network weighted by the driver’s sight line and the distance to the object to extract the spatial graph convolution features of the driver’s sight line direction, such as Figure 4 As shown;

[0156] S3-1: The video frame image obtained in step S1-3 is passed through the RCNN network to detect the object position o and its feature f in the image o , using it as a node of the spatial graph convolutional network;

[0157] S3-2: Calculate the weighted line of sight between the driver and the object and use it as the edge of the graph convolutional network;

[0158] S3-2-1: Obtain the head position coordinates p of step S2-1 h , the gaze point coordinate p in step S2-4 g and the position o of the object detected in step S3-1, calculate the driver's direction and the position o of the object j j Angles between directions:

[0159]

[0160] S3-2-2: Calculate the weighted relationship between the driver and the object's sight line:

[0161]

[0162] Among them j represents the position of object j, represents Gaussian distribution, σ g represents the parameters of the gaze-weighted Gaussian distribution,

[0163] S3-3: Calculate the distance weighted relationship between the driver and the object and use it as an edge:

[0164]

[0165] Among them i and o j denote the positions of object i and object j respectively, d(·) denotes the Euclidean distance between object i and object j, represents Gaussian distribution, σ d Represents the parameters of the distance-weighted Gaussian distribution;

[0166] S3-4: Calculate the spatial graph convolution features of the driver’s sight direction;

[0167] S3-4-1: Obtain the object feature f detected by RCNN from step S3-1 o ;

[0168] S3-4-2: Obtain the sight-line weighted relationship in step S3-2 and the distance weighted relationship in step S3-3;

[0169] S3-4-3: Calculate the sight-weighted adjacent matrix:

[0170] A g ={α g (o i ,o j )} i,j ,

[0171] where α g (o i ,o j ) represents object o i With object o j The weighted relationship of sight lines between

[0172] S3-4-4: Calculate the sight-weighted spatial graph convolution features:

[0173]

[0174] Among them, A g represents the gaze-weighted adjacency matrix, W g represents the parameters of graph convolution, The dimension is 1×4096;

[0175] S3-4-5: Calculate the distance-weighted adjacency matrix:

[0176] A d ={α d (o i ,o j )} i,j ,

[0177] where α d (o i ,o j ) represents object o i With object o j The distance-weighted relationship between

[0178] S3-4-6: Calculate distance-weighted spatial graph convolution features:

[0179]

[0180] Among them A d Represents the distance-weighted proximity matrix, W d represents the parameters of graph convolution, The dimension is 1×4096;

[0181] S3-4-7: Calculate the spatial graph convolution feature f of the driver's line of sight fusion :

[0182]

[0183] in represents the serial operation, φ fc Represents a fully connected layer with an output dimension of 1024;

[0184] S4: Spatial Graph Convolutional Network Training Based on Contrastive Learning;

[0185] S4-1: Calculate driver behavior score;

[0186] S4-1-1: Constructing the classifier φ cls , which contains a temporal convolution, ReLU activation function and Dropout layer;

[0187] S4-1-2: Obtain the spatial graph convolution feature f of the driver's sight direction through step S3-4 fusion ;

[0188] S4-1-3: Obtain segment-level behavior prediction scores:

[0189] s n,m =φ cls (f fusion ),

[0190] where s n；m represents the behavior prediction score of the mth segment in the nth video, n = 1, 2...N, m = 1, 2...M;

[0191] S4-2: Difficult background segment mining;

[0192] S4-2-1: Add a threshold to the action classification result score obtained in step S4-1-4, and use a step function to generate a binary sequence, where 1 corresponds to action and 0 corresponds to background:

[0193]

[0194] Where ε(·) represents the Heaviside step function, b is 0.5, represents the threshold, and when s n；m >b, If it is 1, it indicates an action clip, otherwise it is 0, indicating a background clip;

[0195] S4-2-2: Apply binary dilation operation (·;*) to the behavior prediction score + , expand the time range of the behavior interval and express the expansion operation as and Where MASK is a large mask and mask is a small mask;

[0196] S4-2-3: Computationally difficult background clip set:

[0197]

[0198] represents the set of difficult background fragments, that is, When it is 0;

[0199] S4-2-4: From the difficult background fragment set Select difficult background clips

[0200]

[0201] in, yes The set of K fragment features in , represents the features of the mth segment in the nth video, yes subset of , where K is a hyperparameter, if If the number of mid-segments is less than K, the reset sampling method is used;

[0202] S4-2-5: Apply binary erosion operation (·;*) to the behavior prediction score - , narrowing the time range of the behavior interval and expressing the erosion operation as and Among them, MASK is a large mask, and mask is a small mask;

[0203] S4-2-6: Collection of fragments of computationally difficult behaviors:

[0204]

[0205] in The set of segments representing difficult behaviors, i.e. When it is 1;

[0206] S4-2-7: From the difficult behavior episode collection Difficult behavior feature set

[0207]

[0208] in, yes The set of K fragment features in , yes subset of , where K is a hyperparameter, if If the number of segments is less than K, the reset sampling method is used;

[0209] S4-3: Simple fragment mining;

[0210] S4-3-1: Score the segment-level behavior prediction score s in step S4-1-3 n；m The index is sorted in descending order, and the index in step S4-2-3 is removed during the sorting process. and step S4-2-5 The corresponding fragments get the descending index

[0211] S4-3-2: by descending index Selecting a simple behavioral feature set

[0212]

[0213] Where K is a hyperparameter, represent The first K indices of ;

[0214] S4-3-3: Score the segment-level behavior prediction score s in step S4-1-4 n；m The index is sorted in ascending order, and the index in step S4-2-3 is removed during the sorting process. and step S4-2-5 The corresponding fragments get the ascending index

[0215] S4-3-4: indexed in ascending order Selecting a simple background feature set

[0216]

[0217] Where K is a hyperparameter, represent The first K indices of ;

[0218] S4-4: Calculate behavioral loss function;

[0219] S4-4-1: Execute step S4-1-3 to obtain the behavior prediction score s of the segment n,m ;

[0220] S4-4-2: Take s n,m The top k sequences with the highest scores in , and calculate their average value as a n ;

[0221] S4-4-3; the average value a n Through a Softmax function, we get the video-level behavior prediction s n ;

[0222] S4-4-4: Obtaining the true value labels of driver training video categories Calculate the behavior loss function:

[0223]

[0224] S4-5: Calculate the segment contrast loss function;

[0225] S4-5-1: Execute step S4-2-7 to obtain the difficult behavior feature set for Each behavior feature f in step S4-3-2 is randomly selected from the simple behavior feature set Select a behavior feature f + , simple background feature set Select K background features f from - , calculate the loss function:

[0226]

[0227] where f T represents the transposed matrix of f, τ is a hyperparameter, where represents the kth background feature, k = 1, 2...K;

[0228] S4-5-2: Execute step S4-2-4 to obtain the difficult background feature set for Each behavioral feature f in step S4-3-2 is randomly selected from the simple background feature set Select a background feature f from + , simple behavioral feature set Select K background features f from - , calculate the loss function:

[0229]

[0230] where f T represents the transposed matrix of f, τ is a hyperparameter, where represents the kth background feature, k = 1, 2...K;

[0231] S4-5-3: Calculate the segment contrast loss function

[0232]

[0233] in Indicates that Each behavioral feature f in the set of simple behavioral features is randomly selected from Select a behavior feature f + , simple background feature set Select K background features f from - , calculate the mean expected value operation of the loss function; Indicates that Each behavioral feature f in the Select a background feature f + , simple behavioral feature set Select K background features f from - , calculate the mean expected value operation of the loss function;

[0234] S4-6: Calculate the total loss function L of the contrastive learning network for all training videos total :

[0235]

[0236] The balance factor λ is 0.01;

[0237] S4-7: Use the total loss function calculated in step S4-6 as the gradient, and use the back-propagation algorithm to solve the parameters of the model in turn. During the back-propagation process, the parameters that need to be solved include: the parameters of the three fully connected layers in step S2-4-1, the parameters of the two fully connected layers in step S2-4-2, the parameters of the graph convolution in steps S3-4-4 and S3-4-6, and the classifier parameters in step S4-1-3; when the training is completed, the optimal parameter set of the model is obtained.

[0238] S5: Driver distraction detection based on contrastive learning;

[0239] S5-1: Execute step S1 to obtain a test video of the driver during driving, and perform segmentation and sampling processing to obtain video frames;

[0240] S5-2: Execute step S2 on the obtained video frame to extract the driver's gaze point coordinates p g ;

[0241] S5-3: Execute step S3 to obtain the spatial graph convolution feature f of the driver's sight direction fusion ;

[0242] S5-4: Convolutional feature f is obtained from the spatial graph fusion Use step S4-1-3 to get the segment-level behavior prediction score s n；m ;

[0243] S5-5: The obtained segment-level behavior prediction score s n；m Use step S4-4-3 to get the video-level behavior prediction score s n ,

[0244] S5-6: If the predicted score is s n If the value is greater than the threshold of 0.6, it indicates distracted driving; otherwise, it does not indicate distracted driving.

Claims

1. A method for detecting driver distraction based on gaze direction contrast learning, characterized by: The specific steps include: S1: Process the driver's head video to obtain video frames; S2: Build a gaze point extraction network to extract the driver's gaze point coordinates; S3: Construct a spatial graph convolutional network weighted by the driver’s sight line and the distance to the object to extract the spatial graph convolution features of the driver’s sight line direction; S4: Spatial Graph Convolutional Network Training Based on Contrastive Learning; S5: Driver distraction detection based on contrastive learning; Step S1 specifically includes the following steps: S1-1: During driving, use the infrared camera located in front of the driver to obtain the driver's head video V n and its true value label Where n=1,2...N, represents the number of videos, 0 means non-distracted driving, 1 indicates distracted driving; S1-2: Each video V obtained in step S1-1 n Divide into multiple non-overlapping snippets n,m , where m = 1, 2...M, represents the number of fragments; S1-3: For each snippet n,m Perform random sampling to get a video frame n；m ; Step S2 specifically includes the following steps: S2-1: The video frame obtained in step 1-3 is n；m Through the head position extraction model, the head position coordinate p is obtained h ; S2-2: Calculate the loss function of the head position coordinates; where p h_gt represents the true value of the head coordinate, d(p h ,p h_gt ) indicates p h and p h_gt The Euclidean distance between S2-3: Extract the driver’s binocular features; S2-3-1: Crop the video frame obtained in S1-2, extract the left and right eye areas, and resize them to 36×60; S2-3-2: The normalized left eye area and right eye area are passed through the parallel ResNet-18 network respectively, and the obtained left eye feature is represented as f l , the right eye feature is represented as f r , whose dimensions are all 1×224; S2-3-3: Set the left eye feature f l and right eye feature f r The two layers are connected in series through a fully connected layer with an output dimension of 256 to obtain the driver’s binocular features f e ; S2-4: Predict gaze point coordinates p g ; S2-4-1: The head position coordinates p obtained in step S2-1 are h , through three fully connected layers with an output dimension of 256, the head position feature f is obtained h ; S2-4-2: The binocular feature f obtained in step S2-3-3 is e and the head position feature f obtained in step S2-4-1 h Connected in series, through two fully connected layers with output dimensions of 256 and 2 respectively, the gaze point coordinate p is obtained g ; S2-5: Calculate the loss function of the gaze point coordinates; S2-5-1: Calculate the normalized vector from the driver's head to the gaze point: S2-5-2: Calculate the loss function of the gaze point coordinates: where p g represents the predicted gaze point coordinates, p g_gt represents the true value of the gaze point coordinate, p h Represents the predicted head position coordinates.

2. The method for detecting driver distraction based on gaze direction contrast learning according to claim 1, characterized in that: Step S3 specifically includes the following steps: S3-1: The video frame image obtained in step S1-3 is passed through the RCNN network to detect the object position o and its feature f in the image o , using it as a node of the spatial graph convolutional network; S3-2: Calculate the weighted line of sight between the driver and the object and use it as the edge of the graph convolutional network; S3-2-1: Obtain the head position coordinates p of step S2-1 h , gaze point coordinate p in step S2-4 g and the position o of the object detected in step S3-1, calculate the driver's direction and the position o of the object j j Angles between directions: S3-2-2: Calculate the weighted relationship between the driver and the object's sight line: Among them j represents the position of object j, N(·) represents Gaussian distribution, σ g represents the parameters of the gaze-weighted Gaussian distribution; S3-3: Calculate the distance weighted relationship between the driver and the object and use it as an edge: Among them i and o j denote the positions of object i and object j respectively, d(·) denotes the Euclidean distance between object i and object j, N(·) denotes the Gaussian distribution, σ d Represents the parameters of the distance-weighted Gaussian distribution; S3-4: Calculate the spatial graph convolution features of the driver’s sight direction; S3-4-1: Obtain the object feature f detected by RCNN from step S3-1 o ; S3-4-2: Obtain the sight-line weighted relationship in step S3-2 and the distance weighted relationship in step S3-3; S3-4-3: Calculate the sight-weighted adjacent matrix: A g ={a g (the i ,o j )} i,j , where α g (o i ,o j ) represents object o i With object o j The weighted relationship of sight lines between S3-4-4: Calculate the sight-weighted spatial graph convolution features: Among them, A g represents the gaze-weighted adjacency matrix, W g represents the parameters of graph convolution, The dimension is 1×4096; S3-4-5: Calculate the distance-weighted adjacency matrix: A d ={a d (the i ,o j )} i,j , where α d (o i ,o j ) represents object o i With object o j The distance-weighted relationship between S3-4-6: Calculate distance-weighted spatial graph convolution features: Among them A d Represents the distance-weighted proximity matrix, W d represents the parameters of graph convolution, The dimension is 1×4096; S3-4-7: Calculate the spatial graph convolution feature f of the driver's line of sight fusion : in represents the serial operation, φ fc Represents a fully connected layer with an output dimension of 1024.

3. The method for detecting driver distraction based on gaze direction contrast learning according to claim 2, characterized in that: Step S4 The specific steps include: S4-1: Calculate driver behavior score; S4-1-1: Constructing the classifier φ cls , which contains a temporal convolution, ReLU activation function and Dropout layer; S4-1-2: Obtain the spatial graph convolution feature f of the driver's sight direction through step S3-4 fusion ; S4-1-3: Obtain segment-level behavior prediction scores: s n,m =φ cls (f fusion ), where s n；m represents the behavior prediction score of the mth segment in the nth video, n=1,2...N, m=1,2...M; S4-2: Difficult background segment mining; S4-2-1: Add a threshold to the action classification result score obtained in step S4-1-4, and use a step function to generate a binary sequence, where 1 corresponds to action and 0 corresponds to background: Where ε(·) represents the Heaviside step function, b is 0.5, represents the threshold, and when s n；m >b, If it is 1, it indicates an action clip, otherwise it is 0, indicating a background clip; S4-2-2: Apply binary dilation operation (·;*) to the behavior prediction score + , expand the time range of the behavior interval and express the expansion operation as and Where MASK is a large mask and mask is a small mask; S4-2-3: Computationally difficult background snippet set: represents the set of difficult background fragments, that is, When it is 0; S4-2-4: From the difficult background fragment set Select difficult background clips in, yes The set of K fragment features in , represents the features of the mth segment in the nth video, yes subset of , where K is a hyperparameter, if If the number of segments is less than K, the reset sampling method is used; S4-2-5: Apply binary erosion operation (·;*) to the behavior prediction score - , narrowing the time range of the behavior interval and expressing the erosion operation as and Among them, MASK is a large mask, and mask is a small mask; S4-2-6: Collection of fragments of computationally difficult behaviors: in The set of segments representing difficult behaviors, i.e. When it is 1; S4-2-7: From the difficult behavior episode collection Difficult behavior feature set in, yes The set of K fragment features in , yes subset of , where K is a hyperparameter, if If the number of segments is less than K, the reset sampling method is used; S4-3: Simple fragment mining; S4-3-1: Score the segment-level behavior prediction score s in step S4-1-3 n；m The index is sorted in descending order, and the index in step S4-2-3 is removed during the sorting process. and step S4-2-5 The corresponding fragments get the descending index S4-3-2: by descending index Selecting a simple behavioral feature set Where K is a hyperparameter, represent The first K indices of ; S4-3-3: Score the segment-level behavior prediction score s in step S4-1-4 n；m The index is sorted in ascending order, and the index in step S4-2-3 is removed during the sorting process. and step S4-2-5 The corresponding fragments get the ascending index S4-3-4: indexed in ascending order Selecting a simple background feature set Where K is a hyperparameter, represent The first K indices of ; S4-4: Calculate behavioral loss function; S4-4-1: Execute step S4-1-3 to obtain the behavior prediction score s of the segment n,m ; S4-4-2: Take s n,m The top k sequences with the highest scores in , and calculate their average value as a n ; S4-4-3; the average value a n Through a Softmax function, we get the video-level behavior prediction s n ; S4-4-4: Obtaining the true value labels of driver training video categories Calculate the behavior loss function: S4-5: Calculate the segment contrast loss function; S4-5-1: Execute step S4-2-7 to obtain the difficult behavior feature set for Each behavior feature f in step S4-3-2 is randomly selected from the simple behavior feature set Select a behavior feature f + , simple background feature set Select K background features f from - , calculate the loss function: where f T represents the transposed matrix of f, τ is a hyperparameter, where represents the kth background feature, k=1,2...K; S4-5-2: Execute step S4-2-4 to obtain the difficult background feature set for Each behavioral feature f in step S4-3-2 is randomly selected from the simple background feature set Select a background feature f from + , simple behavioral feature set Select K background features f from - , calculate the loss function: where f T represents the transposed matrix of f, τ is a hyperparameter, where represents the kth background feature, k = 1, 2...K; S4-5-3: Calculate the segment contrast loss function in Indicates that Each behavioral feature f in the set of simple behavioral features is randomly selected from Select a behavior feature f + , simple background feature set Select K background features f from - , calculate the mean expected value operation of the loss function; Indicates that Each behavioral feature f in the Select a background feature f from + , simple behavioral feature set Select K background features f from - , calculate the mean expected value operation of the loss function; S4-6: Calculate the total loss function L of the contrastive learning network for all training videos total : The balance factor λ is 0.01; S4-7: Use the total loss function calculated in step S4-6 as the gradient, and use the back-propagation algorithm to solve the parameters of the model in turn. During the back-propagation process, the parameters that need to be solved include: the parameters of the three fully connected layers in step S2-4-1, the parameters of the two fully connected layers in step S2-4-2, the parameters of the graph convolution in steps S3-4-4 and S3-4-6, and the classifier parameters in step S4-1-3; when the training is completed, the optimal parameter set of the model is obtained.

4. The method for detecting driver distraction based on gaze direction contrast learning according to claim 3, characterized in that: Step S5 specifically includes the following steps: S5-1: Execute step S1 to obtain a test video of the driver during driving, and perform segmentation and sampling processing to obtain video frames; S5-2: Execute step S2 on the obtained video frame to extract the driver's gaze point coordinates p g ; S5-3: Execute step S3 to obtain the spatial graph convolution feature f of the driver's sight direction fusion ; S5-4: Convolutional feature f is obtained from the spatial graph fusion Use step S4-1-3 to get the segment-level behavior prediction score s n；m ; S5-5: The obtained segment-level behavior prediction score s n；m Use step S4-4-3 to get the video-level behavior prediction score s n , S5-6: If the predicted score is s n If the value is greater than the threshold of 0.6, it indicates distracted driving; otherwise, it does not indicate distracted driving.

Images