A method for identifying dangerous driving actions

Abstract

This invention discloses a method for identifying dangerous driving maneuvers. This method extracts human skeleton posture features and image abstract features from preprocessed driver images and fuses these features to achieve classification and identification of dangerous driver maneuvers in video images. Through preprocessing steps such as illumination compensation, image segmentation, and data enhancement, the method improves the accuracy of driver maneuver recognition under varying lighting conditions and posture changes. It then fuses motion features based on the two-dimensional human skeleton with features extracted from the VGG16 and ResNet18 models to obtain multiple driver maneuver feature representations. Finally, it utilizes a fully connected network for classification, enabling accurate identification of dangerous driver maneuvers.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision image processing, and in particular relates to a method for identifying dangerous driving actions. Background Art

[0002] Traffic accident statistics show that the number of accidents caused by driver errors is higher than the number of accidents caused by environmental and vehicle factors. Therefore, reducing drivers' illegal operations and dangerous driving behaviors can effectively reduce the occurrence of traffic accidents. Therefore, technical research on identifying dangerous driving behaviors is needed to reduce the occurrence of such accidents and protect people's lives.

[0003] The literature [Abdu Gumaei, Mabrook Al-Rakhami, Mohammad Mehedi Hassan, et al. Adeep learning-based driver distraction identification framework over edgecloud on Journal [J]. Neural Computing and Applications, 2020 (1): 1-16] proposed a framework based on a CNN model to detect driver distraction. It used a deep convolutional neural network (CDCNN) model and a fine-tuned model of VGG-16 for training respectively, but the accuracy of some behaviors was not high, and there were still certain errors when classifying similar postures. Reference [Jegham I, Khalifa A, Alouani I, et al. Soft Spatial Attention-Based Multimodal Driver Action Recognition Using Deep Learning [J]. IEEE Sensors, 2021, 21 (2): 1918-1925] uses the grayscale value of the depth image to highlight the driver's action, eliminates the influence of highly cluttered dynamic background, and uses the VGG-16 model combined with the Long Short Term Memory Networks (LSTM) model to extract and classify action features on the time stream. The depth image-based method is highly dependent on the grayscale value analysis used to highlight the driver's action. However, if there is noise or inaccurate grayscale values ​​in the depth image, or in dark scenes or scenes with particularly strong lighting, it may lead to incorrect action classification or feature extraction. The paper [Guo Keyou, Wang Pengshuo, Shi Peipeng, et al. A New Partitioned Spatial–TemporalGraph Attention Convolution Network for Human Motion Recognition [J]. Applied Sciences, 2023, 13(3): 1647-1647] proposed a neural network model for human motion recognition called NEW-STGCN-CA. The model is based on the spatiotemporal graph convolutional network (ST-GCN). The network contains a new partitioning strategy and coordinated attention (CA) mechanism. By integrating the CA attention mechanism model, the network can focus on information related to the input, ignore unnecessary information, and prevent information loss.This model mainly focuses on the propagation and local dependencies of spatiotemporal data, and has weak modeling capabilities for long-term dependencies and global relationships. It is also affected by multiple hyperparameters, including the input time step, the size of the convolution kernel, the number of graph convolution layers, etc.

[0004] Thanks to the rapid increase in computing power and the continued development of neural network algorithm models, most solutions for identifying dangerous driver actions in common scenarios have achieved relatively excellent performance in terms of accuracy, real-time speed, and robustness. However, as application scenarios continue to broaden and expand, most existing solutions based on deep learning neural networks face the following common issues that still need to be addressed: the performance of models recognizing the same action can vary significantly under different lighting conditions or under highly cluttered and changing backgrounds. In dark scenes, scenes with extremely strong lighting, or scenes with a very cluttered and constantly changing driver background, the recognition performance of most current solutions is unsatisfactory, and no paper has yet proposed a universal recognition model that works under different lighting conditions and backgrounds.

[0005] In summary, driver motion recognition is crucial for accurately assessing a driver's driving state in road traffic safety. While various methods have been proposed, practical applications are limited by factors such as lighting conditions and posture variations. Therefore, a method and technology that can accurately identify driver motion under varying lighting conditions and posture variations is urgently needed. Summary of the Invention

[0006] In view of the above, the present invention provides a method for identifying dangerous driving actions, which realizes the classification and identification of dangerous driving actions of the driver in the video image by extracting human skeleton posture features and image abstract features from the preprocessed driver image and performing feature fusion.

[0007] A method for identifying dangerous driving maneuvers comprises the following steps:

[0008] (1) Obtain an image dataset of vehicle driving behavior, preprocess the images in the dataset, and manually classify and label the driving actions, and then divide all the images in the dataset into a training set and a test set;

[0009] (2) Constructing a network model for dangerous driving action recognition, which includes:

[0010] The GCN feature extraction module is used to extract the skeleton point information of human body movements from the input image, and then calculate the driver's movement features based on the skeleton point information;

[0011] ResNet feature extraction module, used to extract driver image features D1 focusing on overall structure and semantic information from the input image;

[0012] The VGG feature extraction module is used to extract the driver image features D2 focusing on local details and texture information from the input image;

[0013] Multimodal feature fusion module, used to fuse the driver action features with the driver image features D1 and D2 to obtain a fused feature vector;

[0014] a classification prediction module, which classifies and predicts the driving action in the input image based on the fused feature vector and outputs the confidence level of the driving action in each category;

[0015] (3) Using the training set images to train the above network model;

[0016] (4) Input the image to be tested into the trained network model to predict the category of the driving action in the image.

[0017] Furthermore, the preprocessing process in step (1) includes illumination compensation, image segmentation and data enhancement; the illumination compensation is to use a histogram equalization method based on the HSV color space to perform illumination compensation on the image; the image segmentation is to use a dense convolutional network to achieve the segmentation of the character part and the background part in the image; the data enhancement is to generate a new image by performing a series of transformation operations on the existing image, thereby obtaining more image sample data.

[0018] Furthermore, the specific implementation method of the GCN feature extraction module is as follows: first, the skeleton point information of the human body movement is extracted from the input image, including the coordinate position of each joint point and the connection relationship between the joint points, and displayed in the form of a graph in a low-dimensional space to obtain a human skeleton graph; then, the node features and the edge features represented by the adjacency matrix are extracted from the skeleton graph, and the calculation of the adjacency matrix is ​​modified, and then the node features and edge features are input into a graph convolutional neural network composed of a cascade of multiple GCN (Graph Convolutional Network) modules. Finally, the output of the graph convolutional neural network is superimposed with the node features and input into the attention mechanism module to learn the feature weights of different sub-regions, and finally the driver action features are output.

[0019] Furthermore, the modification calculation method of the adjacency matrix is ​​as follows: first, the human skeleton diagram is divided into 6 sub-regions, namely, head, trunk, left arm, right arm, left leg and right leg. If the skeleton node v i and v j If there is a connection between them and they belong to different sub-regions, the corresponding element value in the adjacency matrix is ​​set to 0; if the skeleton node v i and v jWhen there is a connection between them and they belong to the same sub-region, the corresponding element value in the adjacency matrix is ​​set to 1.

[0020] Furthermore, the ResNet feature extraction module adopts the VGG16 model structure, which is composed of 13 convolutional layers and 3 fully connected layers connected in sequence, and some convolutional layers are separated by maximum pooling layers.

[0021] Furthermore, the VGG feature extraction module adopts the ResNet18 model structure, which is composed of 16 convolutional layers and 2 fully connected layers connected in sequence, and residual connections are used between the convolutional layers.

[0022] Furthermore, the classification prediction module adopts a fully connected neural network, fuses the feature vector input into the network for classification, and outputs an n-dimensional vector, which represents the classification prediction result of the driving action in the input image, where n is the number of categories. The output is then converted into a probability distribution using the softmax function, which represents the probability value of each action category, with a value between 0 and 1.

[0023] Furthermore, the specific implementation process of step (3) is as follows:

[0024] 3.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer;

[0025] 3.2 Input the training set images into the model in batches, forward propagate the model output to obtain the corresponding classification prediction results, and calculate the loss function L between the classification prediction results and the category labels;

[0026] 3.3 Based on the loss function L, the optimizer is used to iteratively update the model parameters through the gradient descent method until the loss function L converges and the training is completed.

[0027] Furthermore, the optimizer adopts the AdamW optimizer in pytorch, and the learning rate is adjusted using the cosine annealing algorithm.

[0028] Furthermore, the loss function L is expressed as follows:

[0029]

[0030] Where: y i is the classification prediction result corresponding to the i-th input image in the current batch, p i is the category label corresponding to the i-th input image in the current batch, and N is the number of input images in the current batch.

[0031] The beneficial technical effects of the present invention are as follows: through preprocessing steps such as illumination compensation, image segmentation and data enhancement, the recognition accuracy of the driver's actions under different lighting conditions and posture changes is improved; the action features based on the two-dimensional human skeleton and the features extracted by the VGG16 and ResNet18 models are fused to obtain a variety of driver action feature representations; and the fully connected network is used for classification to achieve accurate recognition of the driver's dangerous actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the training process of the dangerous driving action recognition method of the present invention.

[0033] Figure 2 Schematic diagram of the test flow of the dangerous driving action identification method of the present invention.

[0034] Figure 3 The driver images before and after segmentation.

[0035] Figure 4 The human skeleton diagram before and after sub-region division.

[0036] Figure 5 Schematic diagram of the process of GCN action feature extraction in the present invention. DETAILED DESCRIPTION

[0037] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] This embodiment uses the State Farm dataset, which is a public dataset on the Kaggle platform. The dataset includes 22,424 images for training and 79,726 images for testing. The size of each image in the dataset is 640×480. The dataset includes images of 26 drivers, and divides the drivers' driving actions into 10 categories, namely safe driving, texting with the right hand, making phone calls with the right hand, texting with the left hand, making phone calls with the left hand, operating the radio, drinking, looking behind, fixing hair or makeup, and talking to passengers. This embodiment marks the ten categories of actions as c0 to c9, and selects images for training and testing from the dataset in a ratio of about 8:2. At the same time, in order to avoid interference, this embodiment performs data screening processing so that drivers who have appeared in the training set will not appear in the test set. The present invention is divided into two parts. First, the basic model used is pre-trained on the training set (such as Figure 1 ), and then test (as Figure 2 shown).

[0039] This implementation uses the GCN model, VGG16 model, and ResNet18 model as basic models. Before inputting the model, the color image is reduced to a size of 256×256, and a sub-image of 224×224 size is randomly cropped from the image as input.

[0040] Training phase

[0041] Step 1: Preprocess the driver image. When taking images of the driver, problems such as uneven lighting and cluttered background often occur, which ultimately lead to a significant decrease in image quality and affect the recognition accuracy. Therefore, after obtaining the input image, preprocessing is required to eliminate redundant information in the image and enhance detectability and integrity.

[0042] 1.1: Light compensation

[0043] This implementation uses a histogram equalization method based on the HSV color space to perform illumination compensation. First, the input image is converted to the HSV color space, where H represents hue, S represents saturation, and V represents brightness. The V channel is then separated, and the mean of the V channel is calculated. The gain coefficient a is calculated from the mean of the V channel. The V channel is then fused with a matrix with all zero pixels of the same size as the V channel using the weight a to achieve illumination compensation. The fusion formula can be expressed as:

[0044] V_fused=(1-a)*V_original+a*V_zero

[0045] Where: V_fused represents the V channel value after fusion, V_original represents the V channel value of the original image, V_zero represents the V channel value of the matrix with all pixels set to 0, and a represents the weight value of V_zero during fusion; finally, the V channel is fused with the H and S channels to achieve light compensation.

[0046] Through light compensation technology, images can be adjusted and enhanced to make the details and colors in the image clearer and more realistic, making them easier to identify.

[0047] 1.2: Image Segmentation

[0048] The purpose of image segmentation is to separate the human part from the background part in the image, so as to facilitate the posture estimation and feature extraction of the segmented human part in subsequent operations, thereby improving the accuracy of action recognition. The network structure used by the present invention to complete the image segmentation module The present invention adopts a dense convolutional network (Dense Convolutional Network, DenseNet) to complete the image segmentation module. DenseNet is a deep neural network model. Its basic structure and principle is to enhance feature transfer and sharing through dense connections, thereby achieving more efficient parameter utilization and better performance. Unlike traditional deep neural networks, the various convolutional layers of DenseNet are not simply forward propagated, but forward transferred directly between the layers, and the output of the previous layer is directly passed to the next layer through dense connections (Dense Connection). Each convolutional layer obtains input from all previous convolutional layers and passes its own feature map to all subsequent convolutional layers. The lth layer will receive all feature maps x0, x1, ..., x0 of the previous l-1 layers. l-1 , and output an x l , x l The expression is as follows:

[0049] x l =H l ([x0,x1,...,x l-1 ])

[0050] Where: x l represents the output of the lth layer, H l Represents a composite function (including convolution, pooling and activation, etc.), [x0,x1,...,x l-1 ] represents the output of the first l-1 layer; the original image and the segmented effect image are compared. Figure 3 shown.

[0051] 1.3: Data Augmentation

[0052] Data augmentation refers to enriching the training dataset by generating new data samples through a series of transformations and expansions on existing data. The purpose of data augmentation is to help the model learn and generalize better, and improve the robustness and performance of the model. The main methods of data augmentation used in this paper are to perform geometric operations such as rotation, inversion, scaling, translation and shearing on the image, and to perform color transformation operations such as brightness, contrast, and saturation on the image to increase data diversity and obtain more sample data. In addition, some random noise is added to the image to increase the robustness of the data. The data augmentation part only exists during the training model process.

[0053] Step 2: Recognize image actions

[0054] 2.1: 2D Skeleton Feature Extraction Based on GCN

[0055] The core idea of ​​2D skeleton feature extraction is to estimate the driver's 2D posture and movement based on the features of the skeleton graph based on the Graph Convolutional Network (GCN).

[0056] First, we input the pre-processed driver image and extract the key skeleton point information of the human body movement, which usually includes the coordinate positions of the joint points and the connection relationship between the joint points. The human skeleton graph is displayed in the form of a graph in a low-dimensional space. The skeleton graph is divided into several regions to obtain more representative local postures and movements. Taking the skeleton graph of the NTU-RGB human skeleton dataset as an example, the human body is divided into 6 sub-regions: head, torso, left arm, right arm, left leg and right leg, denoted as R0 to R5. Figure 4 shown.

[0057] Extract node features from the skeleton graph and edge features based on the adjacency matrix. Modify the calculation of the adjacency matrix, when the skeleton node v i and v j When there is a connection (i.e., adjacent) between them, but they belong to different sub-regions, the corresponding element value of the adjacency matrix is ​​set to 0; when the skeleton node v i and v j When there is a connection (i.e., adjacent) between them and they belong to the same sub-region, the corresponding element value of the adjacency matrix is ​​set to 1; the node feature F i The improved adjacency matrix is ​​input into a GCN network composed of multiple GCN modules, and an attention module is added to learn the feature weights of different sub-regions to obtain the action feature F′ of the driver image. i ; The motion feature extraction process based on skeleton graph is as follows Figure 5 shown.

[0058] 2.2: CNN-based feature extraction

[0059] The present invention extracts driver image features based on two basic models, VGG16 and ResNet18.

[0060] The VGG16 model structure contains 13 convolutional layers and 3 fully connected layers, and is mainly used for image classification tasks. In this implementation, the preprocessed driver image is scaled into a 224×224 3-channel RGB color image and input into the VGG16 model. After 13 convolutional layers and 4 maximum pooling layers, the intermediate layer directly outputs a 4096-dimensional feature vector for characterizing the driver image features.

[0061] The ResNet18 model's basic structure is a residual block, consisting of 18 weighted layers, including 16 convolutional layers and two fully connected layers. Residual connections address the vanishing and exploding gradient problems in deep neural network training by adding the output of the previous layer to the input of the next layer, increasing network depth while making model training more stable. In this implementation, the preprocessed driver image is scaled to a 224×224 3-channel RGB color image and input into the ResNet18 model. After 16 convolutional layers and pooling, a 512-dimensional feature vector is generated to characterize the driver image.

[0062] 2.3: Multimodal Feature Fusion

[0063] The present invention concatenates the two-dimensional skeleton feature vectors of the human body extracted based on GCN, the feature vectors extracted by the VGG16 model, and the feature vectors extracted by the ResNet18 model, and finally inputs them into a fully connected neural network (FC) for classification. The output result is a 10-dimensional vector, which represents the classification prediction result of the network for the input data. The softmax function is then used to convert the output into a probability distribution, which represents the probability value of each category of human action, with a value between 0 and 1.

[0064] Step 3: Optimize the algorithm and calculate the loss function to measure the model performance.

[0065] 3.1: The AdamW optimizer in pytorch is used to optimize the parameters of the neural network model, and the cosine annealing algorithm is used to adjust the learning rate to obtain better training results.

[0066] The main idea of ​​the AdamW algorithm is to add weight decay to the optimization process of the Adam algorithm. Specifically, in the process of calculating the gradient, it first performs L2 regularization on the weight, and then brings the regularized gradient into the optimization process of the Adam algorithm. This can ensure the accuracy of weight decay while avoiding overfitting.

[0067] The cosine annealing algorithm linearly reduces the learning rate from the initial value to a set value, and then periodically adjusts the learning rate in the form of a cosine function. This algorithm can prevent the training from falling into the local minimum by periodically adjusting the learning rate, and can more accurately find the global minimum in the later stages of training.

[0068] 3.2: In terms of loss function, the cross-entropy loss function (Cross-Entropy Loss) is used. The cross-entropy loss function is a loss function used for classification problems. Its essence is to measure the difference between the probability distribution of the model output and the probability distribution of the true label. It can also be regarded as the distance between the model output and the true label. The cross-entropy function formula of this embodiment is as follows:

[0069]

[0070] Where: y i is the output value of sample i, p i is the probability that sample i is correct, L is the value of the loss function, and N is the number of results.

[0071] Table 1 shows the final action recognition accuracy. Directly feeding driver images into the VGG16 model yielded an overall accuracy of approximately 88%. Recognition accuracy for c1 (texting with right hand), c2 (making a phone call with right hand), c6 (drinking), and c8 (fixing hair and makeup) was lower, with c1 (texting with right hand) misidentified as c3 (texting with left hand), and c2 misidentified as c0 (driving safely) and c9 (talking with a passenger). Directly feeding images into the ResNet18 model yielded an overall accuracy of approximately 90%. ResNet18 had lower recognition accuracy for c2 (making a phone call with right hand), c8 (fixing hair and makeup), and c9 (talking with a passenger). The model frequently confused c8 with c6 (drinking) and c9 with c0 (driving safely). Furthermore, the comparison shows that ResNet18 performs better than VGG16.

[0072] Table 1

[0073]

[0074] The overall accuracy of the VGG16 model after preprocessing the image has been greatly improved, but the accuracy of c0 and c9 is still not high. Therefore, the present invention adopts two solutions: fusing different CNN models and fusing GCN-based skeleton features with CNN models to further improve the performance of the model.

[0075] By directly fusing the human skeleton features extracted by the GCN model with the features extracted by the VGG16 model or the ResNet18 model, performance was significantly improved. However, the GCN-VGG16-fused model still had insufficient performance in recognizing actions c2 and c9, and the GCN-ResNet18-fused model still had insufficient performance in recognizing actions c8 and c9.

[0076] The human skeleton feature vector extracted based on the GCN model, the feature vector extracted based on the VGG16 model, and the feature vector extracted based on the ResNet18 model, a total of three types, are directly connected in series and input into a fully connected neural network for driver action classification. It can be seen that the model performance after the fusion of multiple modal features in the present invention is greatly improved, and the overall accuracy can reach about 98%.

[0077] The above description of the embodiments is intended to facilitate understanding and application of the present invention by those skilled in the art. It is apparent that those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without requiring creative effort. Therefore, the present invention is not limited to the above embodiments. Any improvements or modifications made by those skilled in the art based on the disclosure of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for identifying dangerous driving maneuvers, comprising the following steps: (1) Obtain an image dataset of vehicle driving behavior, preprocess the images in the dataset, and manually classify and label the driving actions, and then divide all the images in the dataset into a training set and a test set; The preprocessing process includes illumination compensation, image segmentation, and data enhancement; The illumination compensation is to use a histogram equalization method based on the HSV color space to perform illumination compensation on the image; the image segmentation is to use a dense convolutional network to achieve the segmentation of the human part and the background part in the image; the data enhancement is to generate a new image by performing a series of transformation operations on the existing image, thereby obtaining more image sample data; (2) Constructing a network model for dangerous driving action recognition, which includes: The GCN feature extraction module is used to extract the skeleton point information of human body movements from the input image, and then calculate the driver's movement features based on the skeleton point information; ResNet feature extraction module, used to extract driver image features D1 focusing on overall structure and semantic information from the input image; The VGG feature extraction module is used to extract the driver image features D2 focusing on local details and texture information from the input image; Multimodal feature fusion module, used to fuse the driver action features with the driver image features D1 and D2 to obtain a fused feature vector; a classification prediction module, which classifies and predicts the driving action in the input image based on the fused feature vector and outputs the confidence level of the driving action in each category; (3) Using the training set images to train the above network model; (4) Input the image to be tested into the trained network model to predict the category of the driving action in the image.

2. The method for identifying dangerous driving behavior according to claim 1, characterized in that: The specific implementation method of the GCN feature extraction module is as follows: first, the skeleton point information of the human body movement is extracted from the input image, including the coordinate position of each joint point and the connection relationship between the joint points, and displayed in the form of a graph in a low-dimensional space to obtain a human skeleton graph; then, the node features and the edge features represented by the adjacency matrix are extracted from the skeleton graph, and the calculation of the adjacency matrix is ​​modified, and then the node features and edge features are input into a graph convolutional neural network composed of multiple GCN modules in cascade; finally, the output of the graph convolutional neural network is superimposed with the node features and input into the attention mechanism module to learn the feature weights of different sub-regions, and finally the driver action features are output.

3. The method for identifying dangerous driving behavior according to claim 2, characterized in that: The modification calculation method of the adjacency matrix is ​​as follows: first, the human skeleton diagram is divided into 6 sub-regions, namely, head, trunk, left arm, right arm, left leg and right leg. If the skeleton node v i and v j If there is a connection between them and they belong to different sub-regions, the corresponding element value in the adjacency matrix is ​​set to 0; if the skeleton node v i and v j When there is a connection between them and they belong to the same sub-region, the corresponding element value in the adjacency matrix is ​​set to 1.

4. The method for identifying dangerous driving behavior according to claim 1, characterized in that: The ResNet feature extraction module adopts the VGG16 model structure, which is composed of 13 convolutional layers and 3 fully connected layers connected in sequence, and some convolutional layers are separated by maximum pooling layers.

5. The method for identifying dangerous driving behavior according to claim 1, characterized in that: The VGG feature extraction module adopts the ResNet18 model structure, which is composed of 16 convolutional layers and 2 fully connected layers connected in sequence, and residual connections are used between the convolutional layers.

6. The method for identifying dangerous driving behavior according to claim 1, characterized in that: The classification prediction module uses a fully connected neural network, fuses the feature vector input into the network for classification, and outputs an n-dimensional vector, which represents the classification prediction result of the driving action in the input image, where n is the number of categories. The output is then converted into a probability distribution using the softmax function, representing the probability value of each action category, with a value between 0 and 1.

7. The method for identifying dangerous driving behavior according to claim 1, characterized in that: The specific implementation process of step (3) is as follows: 3.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer; 3.2 Input the training set images into the model in batches, forward propagate the model output to obtain the corresponding classification prediction results, and calculate the loss function L between the classification prediction results and the category labels; 3.3 Based on the loss function L, the optimizer is used to iteratively update the model parameters through the gradient descent method until the loss function L converges and the training is completed.

8. The method for identifying dangerous driving behavior according to claim 7, characterized in that: The optimizer adopts the AdamW optimizer in pytorch, and the learning rate is adjusted using the cosine annealing algorithm.

9. The method for identifying dangerous driving behavior according to claim 7, characterized in that: The expression of the loss function L is as follows: Where: y i is the classification prediction result corresponding to the i-th input image in the current batch, p i is the category label corresponding to the i-th input image in the current batch, and N is the number of input images in the current batch.

Images