A Two-Stage Dual-Channel Distracted Driving Behavior Recognition Method Based on Keypoint Detection

Abstract

This invention belongs to the technical fields of computer vision and machine learning, and relates to a deep learning-based distracted driving behavior recognition technology. Specifically, it provides a two-stage, dual-channel distracted driving behavior recognition method based on keypoint detection to address the poor recognition performance of existing methods. This invention proposes a two-stage recognition network. The first stage uses an Alphapose keypoint detection network to acquire the driver's keypoint information. The second stage uses ResNet-50 as the backbone, designing a heatmap key region map using the keypoint information. A fusion structure is proposed in the dual-channel fusion stage of the original image and the key region map, and spatial and channel attention mechanisms are introduced to enhance the learning of features of interest. Based on the dual-channel network, this invention also introduces a binary classifier module designed using a GCN graph convolutional network model. The binary classifier is used to correct the network prediction results, ultimately effectively improving the recognition performance of distracted driving behavior.

Description

Technical Field

[0001] This invention belongs to the technical fields of computer vision and machine learning, and relates to a deep learning-based distracted driving behavior recognition technology. Specifically, it provides a two-stage dual-channel distracted driving behavior recognition method based on key point detection. Background Technology

[0002] With the improvement of modern people's living standards, the large number of in-vehicle devices and in-car entertainment devices has led to an increasing risk of drivers being distracted, and the frequency of traffic accidents is also rising. In order to ensure safe driving as much as possible and reduce the losses caused by accidents, in-vehicle assistance systems need to be able to accurately identify the driver's driving behavior and provide corresponding reminders to reduce the occurrence of accidents. Distracted driving behavior recognition is an important part of this and has great research and application value.

[0003] Distracted driving behavior recognition faces two main challenges: significant intra-class variability and subtle inter-class variability. Specifically, regarding significant intra-class variability, distracted driving encompasses multiple categories, and even within the same category, variations arise due to factors such as driving environment, clothing, body type, and skin color, introducing substantial noise into the classification network and hindering its ability to focus on information useful for classification. Regarding subtle inter-class variability, the distracted driving dataset often contains multiple behaviors performed by the same person. Because the driver is the same individual, the only distinguishing factor among these different categories of image data is the driver's posture. These postures vary considerably, some significantly, while others are subtle, such as adjusting makeup or drinking water with the right hand. These minute differences undoubtedly increase the difficulty of the classification network. Currently, image-based classification and detection technologies are becoming increasingly sophisticated. Researchers use cameras to capture images of drivers during their driving process and input these images into a trained distraction detection model for detection and recognition. This avoids the cumbersome process of drivers wearing devices to collect physiological signals, and by fixing the camera's shooting area, the detection results are not affected by factors such as the driver's posture and height.

[0004] Currently, methods for recognizing distracted driving behavior mainly include traditional computer vision (CV) algorithms and deep learning algorithms. In early image research, image recognition methods based on HOG (Heterogeneous Origin Group) and SVM (Single-Dimensional Vector Machine) were widely used. These methods first acquire the region of interest (ROI) from the input image, then perform grayscale conversion and filtering to reduce the influence of lighting. Next, the gradient of a specific region is calculated, and by superimposing the gradients of each region, a gradient histogram of the input image is obtained. After obtaining the HOG features, cross-validation is used to find the optimal SVM classifier parameters, followed by classification using the SVM model. However, this method is computationally intensive and has poor real-time performance. Traditional CV algorithms must select features from each image; however, with the dramatic increase in the types and number of images, feature extraction becomes increasingly difficult. TranHiepDinh used depth histogram-based image segmentation to study distracted driving behavior, considering changes in the clustering location or depth value of the ROI exceeding a preset threshold as a distraction. However, this method can only detect features at specific locations of the driver and cannot detect subtle differences between image classes. G. Tanisik et al. proposed contour detection and image segmentation algorithms to detect distracted driving behavior. However, this method extracts a single feature descriptor from the entire image to represent the features of the distracted driving image, which is not suitable for recognizing behaviors with significant variations. Baheti et al. conducted distracted driving detection experiments using the original VGG-16 network and found that the original VGG-16 model had an excessively high fit to the training data. Although it performed well on the training set, it did not show good generalization ability on the test data, and a small portion of images were always misclassified due to unclear category features.

[0005] With the development of computer technology and the improvement of processor processing power, deep convolutional neural networks have achieved excellent results in image processing. Compared with traditional methods, deep learning algorithms have better generalization and robustness. Since the birth of AlexNet, many excellent classification networks have emerged, such as VGG, Inception, and ResNet. Research using convolutional neural networks to analyze driving behavior has become very common, and many researchers have begun to build their own research datasets. Deep learning-based research methods have attracted increasing attention. Koesdwiady et al. created a dataset and used it for experimental model training and testing, proposing an end-to-end deep learning solution for distracted driving image recognition. Their framework utilizes a pre-trained VGG-19 convolutional neural network for feature extraction and adds two fully connected layers to fine-tune the VGG-19 results. Finally, Softmax classification is used to obtain the predicted category. While Koesdwiady et al.'s dataset considered different lighting conditions and camera positions, which improved accuracy, it still failed to account for the small feature differences between different images of the same driver, leading to misclassification. Jing Wang et al. proposed a data augmentation detection method for distracted driving based on driving operation regions. First, the Grad-CAM (gradient-weighted class activation mapping) method is used to display the key feature regions of distracted driving behavior. To effectively remove redundant information in the image and obtain higher classification accuracy, the Faster R-CNN detection model is used to detect driving operation regions for data augmentation. Finally, the augmented dataset is used to complete the distracted driving behavior recognition task through a classification network. Yingcheng Lin et al. proposed a novel lightweight attention-based image classification network (LWANet). To reduce computational cost and trainable parameters, standard convolutional layers are replaced with depthwise separable convolutions, and the classic VGG16 architecture is optimized by 98.16% of the training parameters. Inspired by attention mechanisms in cognitive science, a lightweight attention module (IRAM) is proposed to simulate human attention, extracting more specific features and improving overall accuracy.Yaocong Hu et al. proposed the MSA-CNN (multi-scale attention convolutional neural network) and a new deep learning-based method to extract fine-grained features for driver distraction behavior recognition. Specifically, using ResNet-50 as the backbone, the network was improved in two ways: 1) multi-scale convolutional blocks with different kernel sizes were used to generate hierarchical feature maps, and maximum selection units were used to adaptively combine multi-scale information; 2) an attention mechanism was introduced to learn pixel saliency and channel saliency, enabling the network to strengthen local detail information and suppress global background information. The improved network showed a certain degree of enhanced performance in recognizing distracted driving behavior.

[0006] It is evident that existing distracted driving behavior recognition methods all start from within the classification network, improving recognition accuracy by enhancing attention to local features. However, the self-learning and self-attention of the recognition network are easily affected by scene interference, resulting in attentional bias or inappropriate processing. Therefore, this invention analyzes the limitations and difficulties of existing distracted driving recognition methods and proposes a two-stage recognition network. This network incorporates the driver's key point information acquired in the first stage into the classification network, and based on this, research is conducted on a distracted driving behavior recognition algorithm based on human key point detection. Summary of the Invention

[0007] The purpose of this invention is to address the numerous problems existing in the prior art by providing a two-stage dual-channel distracted driving behavior recognition method based on key point detection. The method employs a one-stage key point detection network to acquire the driver's key point information, introduces the key point information into a classification network, and then introduces a GCN graph convolutional network model from the perspective of graph structure data, ultimately effectively improving the recognition effect of distracted driving behavior.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A two-stage, dual-channel distracted driving behavior recognition method based on key point detection, characterized by the following steps:

[0010] Step 1. Input the original image to be identified into the Alphapose keypoint detection network, and the keypoint detection network outputs a feature matrix containing driver keypoint information;

[0011] Step 2. Based on the driver's key point information, generate a heatmap of key areas using a Gaussian heatmap method;

[0012] Step 3. Construct a dual-channel distracted driving behavior recognition network, using the original image and the Heatmap key region map as dual-channel inputs, and output the initial predicted probability distribution from the dual-channel distracted driving behavior recognition network;

[0013] Step 4. Construct a binary classifier module, taking the feature matrix and the initial predicted probability distribution as dual-channel inputs, and outputting the final predicted probability distribution, where the distracted driving behavior category corresponding to the maximum probability value is the recognition result.

[0014] Furthermore, in step 1, the driver's key points include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left elbow, and right wrist; the driver's key point information includes: horizontal pixel coordinates, vertical pixel coordinates, and confidence level.

[0015] Furthermore, in step 2, the Gaussian heatmap method specifically involves: using a Gaussian distribution function to generate a key region heatmap around the key point that conforms to a Gaussian distribution, centered on the key point, to obtain the key region heatmap for each key point; aligning all key region heatmaps with the original image to obtain a heatmap template with the same size as the original image; and fusing the heatmap template with the original image according to preset weights to obtain the Heatmap key region map.

[0016] Furthermore, in step 3, the dual-channel distracted driving behavior recognition network includes: a feature fusion module, a ResNet-50 module, an attention module, and a fully connected output layer; among which,

[0017] The feature fusion module includes convolutional layers Conv1, Conv2, and Conv3. The original image and the key region map of the heatmap are processed by convolutional layers Conv1 and Conv2 respectively to extract features, resulting in the original feature map and the key region feature map of the heatmap. The original feature map and the key region feature map of the heatmap are then concatenated to obtain the concatenated feature map. The concatenated feature map is then processed by convolutional layer Conv3 to obtain the initial fused feature map.

[0018] The ResNet-50 module consists of four residual modules. The initial fused feature map is input into the ResNet-50 module for further feature extraction, and the ResNet-50 module outputs a secondary fused feature map.

[0019] The attention module includes a spatial attention module SAM and a channel attention module CAM. Among them, the secondary fusion feature map passes through the attention module SAM to obtain a spatial attention response map, and then the spatial attention response map is multiplied by the secondary fusion feature map to obtain a primary enhanced feature map; the primary enhanced feature map passes through the channel attention module CAM to obtain a channel attention response map, and then the channel attention response map is multiplied by the primary enhanced feature map to obtain a secondary enhanced feature map;

[0020] The secondary enhanced feature map passes through a fully connected output layer to obtain a primary predicted probability distribution for the distracted driving behavior category.

[0021] Furthermore, in step 4, the binary classifier module includes a GCN binary classifier module group and a judgment fusion module; among them,

[0022] The GCN binary classifier module group consists of GCN binary classifier models. Each GCN binary classifier model corresponds to a binary classification task for any two types among J types of distracted driving behaviors to be recognized (J is the number of types of distracted driving behaviors to be recognized);

[0023] Search for the maximum value m1 and the second maximum value m2 in the primary predicted probability distribution, select a GCN binary classifier model according to the types of distracted driving behaviors corresponding to the maximum value m1 and the second maximum value m2; input the feature matrix into the selected GCN binary classifier model, and the selected GCN binary classifier model outputs a secondary predicted probability distribution n1 and a second maximum value n2;

[0024] Then input m1 and m2, n1 and n2 into the judgment fusion module together for prediction probability fusion to obtain the final predicted probability distribution.

[0025] Even further, in step 4, the fusion logic of the judgment fusion module is:

[0026] Primary judgment fusion: When m1 < m2 and n1 > n2, or m1 > m2 and n1 < n2, take the primary predicted probability distribution (m1, m2) output by the dual-channel recognition network as the final predicted probability distribution; otherwise, perform secondary judgment fusion;

[0027] Secondary judgment fusion: When |m1 - m2| ≤ k|n1 - n2|, fuse m1 and m2, n1 and n2 to obtain (m1 + n1, m2 + n2) as the final predicted probability distribution; otherwise, take the primary predicted probability distribution (m1, m2) output by the dual-channel recognition network as the final predicted probability distribution; where k is a preset constant.

[0028] Based on the above technical solutions, the beneficial effects of the present invention are:

[0029] This invention proposes a distracted driving behavior recognition method based on human keypoint detection. Research has found that graph structure data formed by connecting driver keypoints according to the human skeleton distribution has a strong correlation with the driver's distracted driving behavior category. Based on this, a two-stage dual-channel distracted driving recognition method is proposed from the perspective of pre-acquiring regions of interest. In the first stage, an Alphapose keypoint detection network pre-trained on the SF3D dataset is used to acquire driver keypoint information. In the second stage, ResNet-50 is used as the backbone. In order to give full play to the role of keypoints, a heatmap key region map is designed using keypoint information. A fusion structure is proposed in the dual-channel fusion stage of the original map and the key region map. Spatial and channel attention mechanisms are introduced from the network internally to enhance the learning of features of interest. Furthermore, based on the dual-channel recognition network of this invention, a graph convolutional network (GCN) model is innovatively introduced, and a recognition method that combines the GCN network and the dual-channel network is proposed. A binary classifier module composed of a corresponding number of GCN binary classification models is constructed, and the maximum and second-largest values ​​of the prediction layer of the dual-channel network are corrected by the binary classifier module, thereby further enhancing the recognition effect.

[0030] Finally, this invention was trained and tested on the SF3D (State Farm Distracted Driver Detection) dataset, which was taken from the Kaggle Distracted Driving Recognition Competition dataset. It contains 10 driving behavior categories: c0 - safe driving, c1 - texting with right hand, c2 - making a phone call with right hand, c3 - texting with left hand, c4 - making a phone call with left hand, c5 - operating the radio control panel, c6 - drinking, c7 - reaching behind, c8 - fixing hair and makeup, and c9 - talking to passengers. Compared to existing technologies, this invention effectively improves the recognition of distracted driving behaviors. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the two-stage dual-channel distracted driving behavior recognition method based on key point detection in this invention.

[0032] Figure 2 This is a schematic diagram of the Heatmap key area map production process in this invention.

[0033] Figure 3 This is a schematic diagram of the dual-channel distracted driving behavior recognition network in this invention.

[0034] Figure 4 This is a schematic diagram of the feature fusion module in this invention.

[0035] Figure 5 This is a schematic diagram of the binary classifier module in this invention.

[0036] Figure 6 This is a flowchart of the fusion logic of the fusion module in this invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0038] This invention provides a two-stage dual-channel distracted driving behavior recognition method based on keypoint detection. Its core innovations are: 1) Constructing a two-stage distracted driving recognition framework based on human keypoint detection, consisting of a one-stage Alphapose keypoint detection network and a two-stage dual-channel distracted driving recognition network; 2) To enhance attention to and learning of regions of interest, the network structure is designed from both external and internal perspectives. The former utilizes the heatmap key region map obtained from the original image and keypoint information to construct the dual-channel recognition network, while the latter introduces spatial and channel attention mechanisms into the network; 3) From the perspective of graph structure data, a binary classifier module based on a GCN network is constructed. The GCN binary classification model in the binary classifier serves as an auxiliary to correct the prediction layer results of the dual-channel network.

[0039] The following is a detailed description of the embodiment: This embodiment provides a two-stage, dual-channel distracted driving behavior recognition method based on human key point detection, the process of which is as follows: Figure 1 As shown, the overall framework mainly includes: a key point detection network (first stage), a dual-channel distracted driving recognition network (second stage), and a binary classifier module, as detailed below.

[0040] (1) Key Point Detection Network

[0041] like Figure 1As shown in Structure 1, the original image is input into the Alphapose keypoint detection network to obtain the driver's keypoint information. In the first stage, the Alphapose keypoint detection network, pre-trained based on the COCO and SF3D datasets, is used to obtain human keypoints. The network obtains keypoints for 17 driver positions: nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles, corresponding to labels 0-16. Analysis shows that the differences between the 10 categories in the SF3D dataset are mainly concentrated in the upper body region, including: nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, and left and right wrists. The lower body regions of different categories show almost no pose differences, so distinguishing based on the lower body region is not very effective. Therefore, this invention only utilizes 11 of the 17 output keypoints: nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, and left and right wrists. The 11 keypoints for the driver are exemplarily represented as follows:

[0042]

[0043] The output of the keypoint detection network is a matrix of dimension C×N×3, where C represents the number of people in the image, N represents the number of keypoints detected, and 3 represents the feature dimension, including the horizontal and vertical pixel coordinates of the keypoints and the confidence score. An example of the feature matrix is ​​as follows:

[0044]

[0045] (2) Dual-channel identification network

[0046] The key point information obtained in the first stage is the coordinate information of the points. However, the region of interest for the identification task needs to be obtained by expanding an appropriate range centered on the key points. Therefore, this invention generates a heatmap key region map based on the key point information using a Gaussian heatmap. The heatmap key region map generation process is as follows: Figure 2 As shown, firstly, a Gaussian distribution function is used to generate a Gaussian distribution heatmap of the key regions surrounding each key point, centered on the key point, thus obtaining a heatmap of the key region for each key point. Then, all key region heatmaps are aligned with the original image to obtain a heatmap template of the same size as the original image, containing multiple Gaussian distribution key regions, as shown. Figure 2 As shown in (b); finally, the heatmap template and the original image are fused according to a preset weight (in this embodiment, the weights of both the heatmap template and the original image are set to 0.5) to obtain a heatmap key region map, as shown in Figure (b). Figure 2 As shown in (c).

[0047] like Figure 1As shown in structure 2, the original image and the heatmap key region map are input into the dual-channel recognition network as two channels; the structure of the dual-channel recognition network is as follows: Figure 3 As shown, it mainly includes a feature fusion module, a ResNet-50 module, and an attention module; the details are as follows:

[0048] The feature fusion module is as follows Figure 4 As shown, it includes three convolutional layers Conv1, Conv2, and Conv3 with a kernel size of 7×7. The original image and the Heatmap key region map are processed by convolutional layers Conv1 and Conv2 respectively for feature extraction, resulting in an original feature map and a Heatmap key region feature map with 64 channels respectively. The original feature map and the Heatmap key region feature map are then concatenated to obtain a concatenated feature map with 128 channels. The concatenated feature map is then processed by convolutional layer Conv3 to obtain an initial fusion feature map with the number of channels reduced to 64.

[0049] The ResNet-50 includes four residual modules. The initial fused feature map is input into the ResNet-50 for further feature extraction, and the ResNet-50 outputs a secondary fused feature map to the attention module.

[0050] The attention module includes a spatial attention module (SAM) and a channel attention module (CAM). The secondary fusion feature map is processed by the attention module SAM to obtain a spatial attention response map, and then the spatial attention response map is multiplied by the secondary fusion feature map to obtain an initial enhanced feature map. The initial enhanced feature map is processed by the channel attention module CAM to obtain a channel attention response map, and then the channel attention response map is multiplied by the initial enhanced feature map to obtain a secondary enhanced feature map. Finally, the secondary enhanced feature map is processed by a fully connected layer to obtain the initial predicted probability distribution for the distracted driving behavior category. Specifically, the spatial attention mechanism first performs dimensionality compression on each feature point along the channel dimension, compressing the original 3D feature map into 2D. Compression methods include max pooling and mean pooling. The two resulting 2D feature maps are concatenated and then processed by a convolutional layer to extract a new 2D feature map, the spatial attention response map. Multiplying the spatial attention response map with the input feature map enhances the region of interest. The channel attention mechanism, from a channel perspective, first downsamples and compresses the feature map of each channel of the input feature map, resulting in a one-dimensional vector with a length equal to the number of channels. This one-dimensional vector effectively weights each channel. Then, the one-dimensional weight vector is learned by a multilayer perceptron to establish connections between different channels. Finally, the channel weight vector attention response map is multiplied with the input feature map to enhance the influence of critical channels and suppress the influence of non-critical channels. See the channel attention mechanism module for details. Figure 3 CAM structure.

[0051] In the dual-channel recognition network, the feature map fused by the feature fusion module enhances the attention of the region of interest by fusing the features of the fusion key region map. After the feature extraction of four residual modules, from the perspective of improving the internal structure of the network, an attention mechanism is introduced, and the learning of the key region is further enhanced by adding attention mechanisms in both the spatial and channel aspects.

[0052] (3) GCN Binary Classifier Module

[0053] Such as Figure 1 The structure 3 in is the binary classifier module, and the specific structure of the binary classifier module is as shown in Figure 5 shown; the main function of the binary classifier module is to use the prediction results of the driver distraction behavior categories based on the GCN binary classification model to assist in correcting the prediction results based on the dual-channel recognition network. The two prediction results start from the perspectives of image and graph structure data respectively, and they complement each other to enhance the recognition effect of distracted driving behavior; specifically:

[0054] The two inputs of the binary classifier module are respectively the initial prediction probability distribution output by the dual-channel recognition network and the feature matrix corresponding to the original image; the binary classifier module selects the binary classification models trained based on these two categories from 45 GCN binary classification models according to the two categories corresponding to the top2 (the maximum value and the second maximum value) in the prediction result of the dual-channel network, denoted as m1 and m2. The feature matrix passes through the selected binary classification model to obtain the secondary prediction probability distribution of this binary classification, denoted as n1 and n2; among them, m1 corresponds to the same category as n1, and m2 corresponds to the same category as n2; it should be noted that: the binary classification model is essentially a GCN network model based on the binary classification task, and the specific structure is well-known technology in the art and will not be elaborated in this invention;

[0055] m1 and m2, n1 and n2 are jointly input into the judgment fusion module for prediction probability fusion, and the fusion logic of the judgment fusion module is as shown in Figure 6 shown, specifically:

[0056] Initial judgment fusion: When m1 < m2 and n1 > n2, or m1 > m2 and n1 < n2, the initial prediction probability distribution (m1, m2) output by the dual-channel recognition network is used as the prediction output of the driver distraction behavior category; otherwise, secondary judgment fusion is performed;

[0057] Secondary judgment fusion: When |m1-m2|≤k|n1-n2|, m1 and m2, n1 and n2 are fused to obtain (m1+n1, m2+n2) as the driver distraction behavior category prediction output; otherwise, the initial prediction probability distribution (m1, m2) output by the dual-channel recognition network is used as the driver distraction behavior category prediction output; where k is a preset constant, and in this embodiment, the optimal value is set to 0.73;

[0058] Finally, the category corresponding to the maximum probability in the driver distraction behavior category prediction output is the recognition result.

[0059] This invention sets the number of binary classification models in the binary classifier module to... In this invention, a binary classification model based on a GCN network is established for each pair of the 10 categories. Regardless of which two categories the top two outputs of the dual-channel prediction layer correspond to, the corresponding binary classification model can be selected. Although this invention introduces 45 GCN binary classification models, only one model is ultimately selected from the 45 models to actually participate in the process of distracted driving behavior recognition. Therefore, it does not increase the overall time complexity of the network. On the contrary, because the GCN binary classifier changes the 10-category classification task to a 2-category classification task, the time complexity becomes smaller.

[0060] The classifier module introduced in this invention primarily corrects the prediction layer output of the dual-channel network. Research revealed significant differences in the recognition accuracy of top1, top2, and top3 when using the dual-channel network recognition method alone in the second stage, with accuracy rates of 92.2%, 97.4%, and 98.0%, respectively. The difference of 5.2% from top1 to top2 indicates that 97.4% of the predicted data correspond to the true class values ​​concentrated in the maximum and second-largest values. Specifically, only 92.2% of the true class values ​​are the maximum, while 5.2% are the second-largest, meaning there is a 5.2% correction space from top1 to top2. Correcting the 5.2% of data in the second-largest position to the maximum value using the classifier can effectively improve the recognition performance. Furthermore, since the two classes corresponding to the top2 prediction result of the dual-channel network can be any pairwise combination of 10 classes, the classifier uses a binary classification model. To cover all possible pairwise class combinations, the number of binary classification models in the classifier is [missing information]. There are 45 in total; although there is a 5.8% correction space from top 1 to top 3, slightly larger than the 5.2% for top 2, a three-classifier is needed for correction, and the number of three-classifier models needs to be increased. The number of models is far greater than the number of binary classifiers, requiring a greater sacrifice in space. Furthermore, while increasing the number of classes J (1≤J≤10) increases the correction space, it also increases the confusion level of the J classifier among the classes, and research has shown that the correction space cannot be fully utilized. Therefore, this invention uses a binary classification model to construct a binary classifier module.

[0061] Based on the aforementioned two-stage, dual-channel distracted driving behavior recognition framework based on human keypoint detection, this invention trains the proposed framework model and verifies the effectiveness of the proposed method through testing. Based on the overall classification accuracy (ACC) and the individual class recognition accuracy (c0-c9), the distracted driving behavior recognition method based on human keypoint detection proposed in this invention is compared with other advanced methods. The comparison methods are mainly one-stage methods, including: PHOG-MLP, PAV-SVM, AlexNet, VGG19, PAV-Hint CNN, Multi-stream CNN, and MSA-CNN. The results of the comparison with other methods are shown in Table 1, and the correspondence between numbers ①-⑩ and specific methods is shown in Table 2. The method proposed in this invention is abbreviated as Ours.

[0062] Table 1 Unit: %

[0063]

[0064] Table 2

[0065]

[0066] Table 1 shows the evaluation metrics, including the recognition accuracy for 10 categories and the overall accuracy. The metrics data for the comparison methods were provided by authoritative journals and papers, with the optimal metric for each item indicated in bold. The results show that the recognition accuracy of this invention is superior to other methods in categories c1-c7, and the overall recognition accuracy is significantly better. Only in three categories is the recognition performance worse than the MSA-CNN method proposed by Yaocong Hu et al., but the overall recognition performance of this invention is better than MSA-CNN. Compared to the MSA-CNN method, the recognition accuracy of this invention is improved by 1.4 points. Therefore, this invention achieves better recognition performance than other methods, thus validating the effectiveness of the proposed method.

[0067] The above description is merely a specific embodiment of the present invention. Any feature disclosed in this specification may be replaced by other equivalent or similar features unless otherwise specified. All disclosed features, or steps in all methods or processes, may be combined in any way except for mutually exclusive features and / or steps.

Claims

1. A two-stage, dual-channel distracted driving behavior recognition method based on key point detection, characterized in that, Includes the following steps: Step 1. Input the original image to be identified into the Alphapose keypoint detection network, and the keypoint detection network outputs a feature matrix containing driver keypoint information; Step 2. Based on the driver's key point information, generate a heatmap of key areas using a Gaussian heatmap method; Step 3. Construct a dual-channel distracted driving behavior recognition network, using the original image and the Heatmap key region map as dual-channel inputs, and output the initial predicted probability distribution from the dual-channel distracted driving behavior recognition network; The dual-channel distracted driving behavior recognition network includes: a feature fusion module, a ResNet-50 module, an attention module, and a fully connected output layer; among which, The feature fusion module includes convolutional layers Conv1, Conv2, and Conv3. The original image and the key region map of the heatmap are processed by convolutional layers Conv1 and Conv2 respectively to extract features, resulting in the original feature map and the key region feature map of the heatmap. The original feature map and the key region feature map of the heatmap are then concatenated to obtain the concatenated feature map. The concatenated feature map is then processed by convolutional layer Conv3 to obtain the initial fused feature map. The ResNet-50 module consists of four residual modules. The initial fused feature map is input into the ResNet-50 module for further feature extraction, and the ResNet-50 module outputs a secondary fused feature map. The attention module includes a spatial attention module (SAM) and a channel attention module (CAM). The secondary fusion feature map is processed by the attention module SAM to obtain a spatial attention response map. The spatial attention response map is then multiplied with the secondary fusion feature map to obtain an initial enhanced feature map. The initial enhanced feature map is processed by the channel attention module CAM to obtain a channel attention response map. The channel attention response map is then multiplied with the initial enhanced feature map to obtain a secondary enhanced feature map. The enhanced feature map is passed through a fully connected output layer to obtain the initial predicted probability distribution of the distracted driving behavior category; Step 4. Construct a binary classifier module, taking the feature matrix and the initial predicted probability distribution as dual-channel inputs, and outputting the final predicted probability distribution, where the distracted driving behavior category corresponding to the maximum probability value is the recognition result.

2. The two-stage dual-channel distracted driving behavior recognition method based on key point detection as described in claim 1, characterized in that, In step 1, the driver's key points include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left elbow and right wrist; the driver's key point information includes: horizontal pixel coordinates, vertical pixel coordinates and confidence level.

3. The two-stage dual-channel distracted driving behavior recognition method based on key point detection as described in claim 1, characterized in that, In step 2, the Gaussian heatmap method specifically involves: using the key point as the center, generating a key region heatmap around the key point that conforms to a Gaussian distribution using a Gaussian distribution function, and obtaining the key region heatmap for each key point; aligning all key region heatmaps with the original image to obtain a heatmap template with the same size as the original image. The heatmap template and the original image are fused according to preset weights to obtain the key area heatmap.

4. The two-stage dual-channel distracted driving behavior recognition method based on key point detection as described in claim 1, characterized in that, In step 4, the binary classifier module includes a GCN binary classifier module group and a judgment fusion module; wherein, The GCN binary classifier module consists of The system consists of several GCN binary classifier models, each corresponding to a binary classification task of any two types among the J types of distracted driving behaviors to be identified. Search for the maximum value m1 and the second maximum value m2 in the initial prediction probability distribution, and select a GCN binary classifier model according to the types of distracted driving behaviors corresponding to the maximum value m1 and the second maximum value m2; input the feature matrix into the selected GCN binary classifier model, and output the secondary prediction probability distribution n1 and the second maximum value n2 by the selected GCN binary classifier model; Then input m1 and m2, n1 and n2 into the judgment fusion module together for prediction probability fusion to obtain the final prediction probability distribution.

5. The two-stage dual-channel distracted driving behavior recognition method based on key point detection as described in claim 4, characterized in that, In step 4, the fusion logic of the judgment fusion module is as follows: Initial judgment fusion: When m1 < m2 and n1 > n2, or m1 > m2 and n1 < n2, take the initial prediction probability distribution (m1, m2) output by the dual-channel recognition network as the final prediction probability distribution; otherwise, perform secondary judgment fusion; Secondary judgment fusion: When |m1 - m2| ≤ k|n1 - n2|, fuse m1 and m2, n1 and n2 to obtain (m1 + n1, m2 + n2) as the final prediction probability distribution; otherwise, take the initial prediction probability distribution (m1, m2) output by the dual-channel recognition network as the final prediction probability distribution; where k is a preset constant.

Images