A driver action classification method based on key point enhancement

Abstract

The present invention discloses a driver action classification method based on key point enhancement. The present invention proposes an efficient channel attention module to generate a distinctive global representation. At the same time, the key point features of the driver are extracted by posture estimation. Considering the diversity of key point information, an adaptive weighted residual bottleneck is designed to make the model weight of the input key point feature dynamic. Then the output of the global branch and the key point branch are fused as the classification result. The present invention can further improve the accuracy of driver behavior recognition and has important application value in the field of traffic safety.

Description

Technical Field

[0001] The invention belongs to the field of image processing and pattern recognition, and in particular relates to a driver action classification method based on key point enhancement. Background Art

[0002] Today, advanced driver assistance systems (ADAS) are being developed to prevent accidents by providing technology that alerts drivers to potential problems and to keep car drivers and occupants safe when an accident occurs. However, even today's latest self-driving cars require drivers to stay focused and be ready to take control of the steering wheel back in an emergency. A Tesla self-driving car crashed into a white truck trailer in Williston, Florida in May 2016, marking the first fatal accident in self-driving car testing. In March 2018, an Uber self-driving car with an emergency backup driver behind the wheel struck and killed a pedestrian in Arizona. In both deaths, the driver could have avoided the crash, but the evidence showed that he was clearly distracted. This also makes the detection of distracted drivers an important part of self-driving cars as well. Therefore, distracted driving detection is essential for further preventive measures. If the vehicle can detect this distraction and then warn the driver, the number of human-caused traffic accidents can be reduced. Summary of the invention

[0003] To solve the above problems, the present invention discloses a driver action classification method based on key point enhancement. The adaptive weighted method of key point enhancement using efficient channel attention used in the invention can explicitly extract discriminative regional features in driver images and realize real-time driver behavior recognition in the test phase.

[0004] To achieve the above object, the technical solution of the present invention is as follows:

[0005] A driver action classification method based on key point enhancement comprises the following steps:

[0006] Step 1: Use the existing StateFarm dataset as the driver behavior image dataset;

[0007] Step 2: Construct a key point enhancement model. The present invention uses ResNet-50 as the network backbone and constructs an efficient channel attention module to learn channel attention and extract global features of the input driving image. Lightweight OpenPose is used to detect the driver's head and hands, and RoI Align is used to extract their features. Then, the key point features are input into the adaptive weighted residual bottleneck to adaptively weight the model parameters. The final classification result is the fusion of the global and key point branch outputs.

[0008] Step 201: For the input driving image, ResNet integrated with an efficient channel attention module is used as a global feature extractor; its design idea is: the global average pooling commonly used in convolutional neural networks is a special case of frequency domain feature decomposition, and it is not enough to use it alone to represent channel attention; therefore, the attention of multi-spectral channels proposed in FcaNet is adopted to increase the use of information in different frequency domains; inspired by ECA-Net, this module makes the current convolution channel interact only with k adjacent channels; the kernel size k of the one-dimensional convolution represents the k neighbors involved in the attention calculation; the channel weights in the module are generated by one-dimensional convolution, where the kernel size k is calculated according to the number of channels C as follows:

[0009]

[0010] The subscript odd indicates that k is an odd number. In the designed module, the relevant parameter k=9 is set for ResNet-50.

[0011] Step 202: Lightweight OpenPose is used to detect the key points of the driver, and then RoI Align is applied to generate fine key point features;

[0012] Step 203: Design an adaptive weighted residual bottleneck based on WeightNet. Specifically, use a 1×1 convolutional layer instead of a fully connected layer, and use a grouped 1×1 convolutional layer instead of a grouped fully connected layer; then use the subnet to generate convolution weights for the residual bottleneck; the adaptive weighted residual bottleneck captures more category-specific information than the ordinary residual bottleneck; considering the computational resource consumption of the adaptive weighting operation, the adaptive weighted residual bottleneck is only used in the final convolution stage;

[0013] Step 204: input the key point features into the adaptive weighted residual bottleneck to output the re-weighted key point features; splice the features output by the two branches and feed them into the final classifier;

[0014] Step 3: Train the key point enhancement model; based on the deep learning framework Tensorflow, use adam to optimize the parameters; the traditional data enhancement methods used include (1) cropping the original image with a random aspect ratio ∈ (0.75, 1.33) and scaling it to 224×224; (2) flipping the image horizontally with a probability of 0.5; (3) using color jitter technology to change the brightness, contrast, saturation, and hue of the image; using APEX mixed precision to accelerate the training of the neural network;

[0015] Step 4: Test the keypoint augmentation model.

[0016] The beneficial effects of the present invention are:

[0017] (1) The present invention constructs an efficient channel attention module to enhance channel attention and extract strong representative features.

[0018] (2) The present invention proposes an adaptive weighted residual bottleneck to adaptively generate convolution weights according to the input key point features.

[0019] (3) The present invention further improves the accuracy of driver behavior recognition and has important application value in the field of traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of the present invention;

[0021] Figure 2 are sample pictures of different driving behaviors in the present invention,

[0022] Figure 3 is a schematic diagram of the key point enhancement model structure in the present invention,

[0023] Figure 4 is a schematic diagram of the structure of an efficient channel attention module in the present invention,

[0024] Figure 5 It is a structural schematic diagram of the adaptive weighted residual bottleneck in the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0026] The driver action classification method based on key point enhancement described in the present invention has the following specific implementation steps:

[0027] Step 1: Use the existing StateFarm dataset as the driver behavior image dataset;

[0028] Step 2: Build a key point enhancement model, Figure 3 is a schematic diagram of the structure of the model; the present invention uses ResNet-50 as the network backbone, constructs an efficient channel attention module to learn channel attention and extract global features of the input driving image; Lightweight OpenPose is used to detect the driver's head and hands, and RoI Align is used to extract their features; then, the key point features are input into the adaptive weighted residual bottleneck, and the model parameters are adaptively weighted; the final classification result is the fusion of the global and key point branch outputs;

[0029] Step 201: For the input driving image, ResNet integrated with an efficient channel attention module is used as a global feature extractor; its structural diagram is as follows Figure 4 As shown in the figure, the design idea is that the global average pooling commonly used in convolutional neural networks is a special case of frequency domain feature decomposition. It is not enough to use it alone to represent channel attention. Therefore, the attention of multi-spectral channels proposed in FcaNet is adopted to increase the use of information in different frequency domains. Inspired by ECA-Net, this module makes the current convolution channel interact only with k adjacent channels. The kernel size k of one-dimensional convolution represents the k neighbors involved in the attention calculation. The channel weights in the module are generated by one-dimensional convolution, where the kernel size k is calculated according to the number of channels C as follows:

[0030]

[0031] The subscript odd indicates that k is an odd number. In the designed module, the relevant parameter k=9 is set for ResNet-50.

[0032] Step 202: Lightweight OpenPose is used to detect the key points of the driver, and then RoI Align is applied to generate fine key point features;

[0033] Step 203: Design an adaptive weighted residual bottleneck based on WeightNet, the structural diagram of which is as follows Figure 5 As shown in the figure; specifically, a 1×1 convolutional layer is used instead of a fully connected layer, and a grouped 1×1 convolutional layer is used instead of a grouped fully connected layer; this subnet is then used to generate convolution weights for the residual bottleneck; the adaptive weighted residual bottleneck captures more category-specific information than the ordinary residual bottleneck; considering the computational resource consumption of the adaptive weighting operation, the adaptive weighted residual bottleneck is only used in the final convolution stage;

[0034] Step 204: input the key point features into the adaptive weighted residual bottleneck to output the re-weighted key point features; splice the features output by the two branches and feed them into the final classifier;

[0035] Step 3: Train the key point enhancement model; based on the deep learning framework Tensorflow, use adam to optimize the parameters; the traditional data enhancement methods used include (1) cropping the original image with a random aspect ratio ∈ (0.75, 1.33) and scaling it to 224×224; (2) flipping the image horizontally with a probability of 0.5; (3) using color jitter technology to change the brightness, contrast, saturation, and hue of the image; using APEX mixed precision to accelerate the training of the neural network;

[0036] Step 4: Test the keypoint augmentation model.

[0037] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A driver action classification method based on key point enhancement, characterized in that: The following steps are involved: Step 1: Use the existing StateFarm dataset as the driver behavior image dataset; Step 2: Build a keypoint enhancement model. Use ResNet-50 as the network backbone and build an efficient channel attention module to learn channel attention and extract global features of the input driving image. Use Lightweight OpenPose to detect the driver's head and hands, and use RoI Align to extract their features. Then, input the keypoint features into the adaptive weighted residual bottleneck to adaptively weight the model parameters. The final classification result is the fusion of the global and keypoint branch outputs. Step 3: Train the key point enhancement model; based on the deep learning framework Tensorflow, use adam to optimize the parameters; the traditional data enhancement methods used include (1) cropping the original image with a random aspect ratio ∈ (0.75, 1.33) and scaling it to 224×224; (2) flipping the image horizontally with a probability of 0.5; (3) using color jitter technology to change the brightness, contrast, saturation, and hue of the image; using APEX mixed precision to accelerate the training of the neural network; Step 4: Test the keypoint augmentation model.

2. The driver action classification method based on key point enhancement according to claim 1 is characterized in that: The key point enhancement model described in step 2 uses ResNet-50 as the network backbone, and builds an efficient channel attention module to learn channel attention and extract global features of the input driving image; Lightweight OpenPose is used to detect the driver's head and hands, and RoI Align is used to extract their features; then, the key point features are input into the adaptive weighted residual bottleneck to adaptively weight the model parameters; the final classification result is the fusion of the global and key point branch outputs; the specific steps are as follows: Step 201: For the input driving image, ResNet integrated with an efficient channel attention module is used as a global feature extractor; its design idea is: the global average pooling commonly used in convolutional neural networks is a special case of frequency domain feature decomposition, and it is not enough to use it alone to represent channel attention; therefore, the attention of multi-spectral channels proposed in FcaNet is adopted to increase the use of information in different frequency domains; inspired by ECA-Net, this module makes the current convolution channel interact only with k adjacent channels; the kernel size k of the one-dimensional convolution represents the k neighbors involved in the attention calculation; the channel weights in the module are generated by one-dimensional convolution, where the kernel size k is calculated according to the number of channels C as follows: The subscript odd indicates that k is an odd number. In the designed module, the relevant parameter k=9 is set for ResNet-50. Step 202: Lightweight OpenPose is used to detect the key points of the driver, and then RoI Align is applied to generate fine key point features; Step 203: Design an adaptive weighted residual bottleneck based on WeightNet. Specifically, use a 1×1 convolution layer to replace the fully connected layer, and use a grouped 1×1 convolution layer to replace the grouped fully connected layer; then use the 1×1 convolution layer and the grouped 1×1 convolution layer to generate convolution weights for the residual bottleneck; the adaptive weighted residual bottleneck captures more category-specific information than the ordinary residual bottleneck; considering the computational resource consumption of the adaptive weighting operation, the adaptive weighted residual bottleneck is only used in the final convolution stage; Step 204: Input the key point features into the adaptive weighted residual bottleneck to output the re-weighted key point features; concatenate the features output by the two branches and feed them into the final classifier.

Images