A method for monitoring motor vehicle drivers through a key point condition network

Abstract

The present invention discloses a method for monitoring a motor vehicle driver through a key point conditional network, which explicitly extracts key point features related to actions. The network includes a global branch and a key point branch. In the global branch, an improved dilated pyramid pooling layer is used to capture multi-scale global context, and in the key point branch, a key point conditional calculation module is proposed to enhance the distinguishability of features. During the classification process, each input image does not share the same fully connected layer parameters, but generates expert weights through conditional calculation under the guidance of key point features, so that the model can adapt to driving images of the corresponding category. The present invention enhances attention by generating key points to scale classification features, thereby improving classification accuracy and robustness, and has important application value in the field of traffic safety.

Description

Technical Field

[0001] The invention belongs to the field of image processing and pattern recognition, and in particular relates to a method for monitoring a motor vehicle driver through a key point condition network. Background Art

[0002] Traffic accidents caused by distracted driving have become a threat to people's lives and property. It is particularly important to effectively identify driving behavior. According to a report by the World Health Organization (WHO), 1.35 million people die in traffic accidents every year around the world. Nearly 3,700 people die every day on the world's roads. "One of the most heartbreaking statistics in this report is that road traffic injuries are the leading cause of death among people aged 5 to 29 years old." The report also shows that the number of deaths has increased year by year, and the most common cause behind these accidents is driver distraction. Using mobile phones while driving is very common among young drivers and novice drivers, which further increases the already high risk of crashes and deaths among these groups. Using a phone while driving increases the likelihood of a crash four times, while texting increases the risk of a crash 23 times. Similarly, the reaction time of drivers using a phone is 50% slower than that of those who do not use a phone. Today, more and more modern vehicles are equipped with advanced driver assistance systems (ADAS) such as stability control, traction control, lane departure warning, adaptive cruise control and anti-lock braking systems. These systems are developed to prevent accidents by providing technology that warns drivers of possible problems and keeps drivers and passengers safe in the event of an accident. Detecting driver inattention is extremely important for additional preventive measures. If the vehicle can detect this distraction and then warn the driver, or if the driver is a professional driver, send a warning message to the headquarters, the number of road accidents can be reduced. Summary of the invention

[0003] To solve the above problems, the present invention discloses a method for monitoring a motor vehicle driver through a key point conditional network. The key point conditional calculation method used in the invention can extract the key point features of the driver and embed them into a classification encoder using conditional calculation, so that it has a gating mechanism at the feature channel level and an example-dependent attention mechanism at the input level, thereby improving the recognition accuracy.

[0004] To achieve the above object, the technical solution of the present invention is as follows:

[0005] A method for monitoring a motor vehicle driver via a critical point condition network comprises the following steps:

[0006] Step 1: The experimental data uses the StateFarm distracted driving dataset;

[0007] Step 2: Construct a key point conditional network. The designed model contains two branches. One branch uses the improved dilated pyramid pooling layer based on ResNet to extract the global features of the driving image. The other branch first extracts the key point features through the posture estimation model, and concatenates the two features and sends them to the key point conditional calculation module, where the generated attention vector re-weights the fused features. This module guides the model to use the key point feature information to emphasize the more distinctive feature channels related to the driving behavior classification, and its output is the final classification representation. The specific description is as follows:

[0008] Step 201: The network model uses ResNet as the backbone, sets the convolution layer in Conv5_x to have a stride of 1 and a dilation rate of 2, and combines an improved dilated spatial pyramid pooling layer to extract global features of the driving image; for the input image, large-scale features encode long-range contexts, small-scale features retain details, and the extraction of multi-scale features is achieved by changing the dilation rate, so multiple dilated convolution branches are merged, namely, dilated spatial pyramid pooling ASPP; the ASPP layer is improved in the present invention to fuse the context in the driving image, and considering the different modes of dilated convolution and pooling operations, two maximum pooling layers of 4×4 and 8×8 are added to enrich the diversity of feature fusion to increase the receptive field; then, an upsampling layer is added to restore the feature size; a 1×1 convolution is added to the first parallel path to retain the original features; in addition, a 1×1 convolution is used in all parallel paths to reduce the number of feature channels to 128; finally, the output features of all parallel paths in the improved ASPP layer are connected in series, and the number of output channels is 768;

[0009] Step 202: Lightweight OpenPose is used in the key point branch of the model to locate the driver's head and hands, and then RoI Align is used to extract key point features from the global features;

[0010] Step 203: Considering the differences in driver behavior categories, a key point condition calculation module is proposed to use key point features to conditionally adjust the fully connected layer; the input f of this module k are the key point features extracted from the input driving image, and the expert weights associated with the input The calculation is as follows:

[0011] r(f k )=S(FC(GAP(f k ))) (1)

[0012] Where S represents the Sigmoid activation function, FC represents the fully connected layer, and GAP represents the global average pooling layer; the fully connected layer maps the key point features indicating the driving action category to n expert weights; the original parameters of the fully connected layer are set to {W1 W 2 , …, W n}; W i As the weight of the i-th category feature, that is, the importance of each dimension feature and its impact on the final score; the score of each category is obtained by weighted summation of the features, and then mapped to probability through Softmax;

[0013] The key point related parameters in the fully connected layer are expressed as {α 1 ·W 1 , α 2 , W 2 , …, α n ·W n ); the final classification feature is calculated as F cls =(α 1 ·W 1 +…+α n ·W n )*F (2)

[0014] Where n is the number of experts, · is the dot product operation, and * is the matrix product operation; each kernel W i The dimension of is still the same as that of the original fully connected layer;

[0015] The weights of the n experts in the fully connected layer depend on the key point features of the driver in the image; these key point features reflect the driving behavior category information and guide different experts to focus on the inputs they are interested in; that is, the weights of the n experts are different for all inputs, where each input can be processed with a different weight; the key point condition calculation module is used to refine the granularity of the condition calculation to the level; the parameters of the fully connected layer used for each driving image in actual prediction are different;

[0016] Step 3: Train the key point conditional network; based on the deep learning framework Tensorflow, use softmax as the classification loss function;

[0017] Step 4: Test the keypoint conditioned network.

[0018] The beneficial effects of the present invention are:

[0019] (1) The present invention explicitly models the key points of the driver in the image, fuses them with the global features, and then uses the proposed key point condition calculation module to guide the classification of the fused features.

[0020] (2) The present invention improves the atrous spatial pyramid pooling layer to enrich the diversity of multi-scale global context feature fusion.

[0021] (3) The present invention further improves the accuracy of driver behavior recognition and has important application value in the field of traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of the present invention;

[0023] Figure 2 are sample pictures of different driving behaviors in the present invention,

[0024] Figure 3 is a schematic diagram of the architecture of the key point conditional network model in the present invention,

[0025] Figure 4 It is a schematic diagram of the structure of the improved atrous space pyramid pooling layer in the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0027] The method for monitoring a motor vehicle driver through a key point condition network described in the present invention has the following specific implementation steps:

[0028] Step 1: The experimental data uses the StateFarm distracted driving dataset;

[0029] Step 2: Construct a key point conditional network model. The designed model contains two branches. One branch uses the improved dilated pyramid pooling layer based on ResNet to extract the global features of the driving image. The other branch first extracts the key point features through the posture estimation model, and concatenates the two features and sends them to the key point conditional calculation module, where the generated attention vector re-weights the fused features. This module guides the model to use the key point feature information to emphasize the more distinctive feature channels related to the driving behavior classification, and its output is the final classification representation. The network structure diagram is shown in the figure. Figure 3 The specific description is as follows:

[0030] Step 201: The network model uses ResNet as the backbone, sets the convolution layer in Conv5_x to have a stride of 1 and a dilation rate of 2, and combines an improved dilated spatial pyramid pooling layer to extract global features of the driving image; for the input image, large-scale features encode long-range contexts, small-scale features retain details, and the extraction of multi-scale features is achieved by changing the dilation rate, so multiple dilated convolution branches are merged, namely, dilated spatial pyramid pooling ASPP; in the present invention, the ASPP layer is improved to fuse the context in the driving image, and the structural schematic diagram of the improved ASPP is shown in FIG. Figure 4As shown in the figure; considering the different ways of operation of dilated convolution and pooling, two maximum pooling layers of 4×4 and 8×8 are added to enrich the diversity of feature fusion to increase the receptive field; then, an upsampling layer is added to restore the feature size; 1×1 convolution is added to the first parallel path to retain the original features; in addition, 1×1 convolution is used in all parallel paths to reduce the number of feature channels to 128; finally, the output features of all parallel paths in the improved ASPP layer are connected in series, and the number of output channels is 768;

[0031] Step 202: Lightweight OpenPose is used in the key point branch of the model to locate the driver's head and hands, and then RoI Align is used to extract key point features from the global features;

[0032] Step 203: Considering the differences in driver behavior categories, a key point condition calculation module is proposed to use key point features to conditionally adjust the fully connected layer; the input f of this module k are the key point features extracted from the input driving image, and the expert weights associated with the input The calculation is as follows:

[0033] r(f k )=S(FC(GAP(f k ))) (1)

[0034] Where S represents the Sigmoid activation function, FC represents the fully connected layer, and GAP represents the global average pooling layer; the fully connected layer maps the key point features indicating the driving action category to n expert weights; the original parameters of the fully connected layer are set to {W 1 , W 2 , …, W n}; W i As the weight of the i-th category feature, that is, the importance of each dimension feature and its impact on the final score; the score of each category is obtained by weighted summation of the features, and then mapped to probability through Softmax;

[0035] The key point related parameters in the fully connected layer are expressed as {α 1 ·W 1 , α 2 · 2 , …, a n ·W n ); the final classification feature is calculated as

[0036] F cls =(α 1 ·W 1 +…+α n ·W n )*F (2)

[0037] Where n is the number of experts, · is the dot product operation, and * is the matrix product operation; each kernel W i The dimension of is still the same as that of the original fully connected layer;

[0038] The weights of the n experts in the fully connected layer depend on the key point features of the driver in the image; these key point features reflect the driving behavior category information and guide different experts to focus on the inputs they are interested in; that is, the weights of the n experts are different for all inputs, where each input can be processed with a different weight; the key point condition calculation module is used to refine the granularity of the condition calculation to the level; the parameters of the fully connected layer used for each driving image in actual prediction are different;

[0039] Step 3: Train the key point conditional network; based on the deep learning framework Tensorflow, use softmax as the classification loss function;

[0040] Step 4: Test the keypoint conditioned network.

[0041] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A method for monitoring a motor vehicle driver through a key point condition network, characterized in that: The following steps are involved: Step 1: The experimental data uses the StateFarm distracted driving dataset; Step 2: Construct a key point conditional network; the designed model contains two branches. One branch uses an improved dilated pyramid pooling layer based on ResNet to extract global features of driving images. The other branch first extracts key point features through a posture estimation model, and concatenates these two features and sends them to a key point conditional calculation module, where the generated attention vector re-weights the fused features. This module guides the model to use key point feature information to emphasize more distinctive feature channels related to driving behavior classification, and its output is the final classification representation. The specific description is as follows: Step 201: The network model uses ResNet as the backbone, sets the convolution layer in Conv5_x to have a stride of 1 and a dilation rate of 2, and combines an improved dilated spatial pyramid pooling layer to extract global features of the driving image; for the input image, large-scale features encode long-range contexts, small-scale features retain details, and the extraction of multi-scale features is achieved by changing the dilation rate, so multiple dilated convolution branches are merged, namely, dilated spatial pyramid pooling ASPP; the ASPP layer is improved in the present invention to fuse the context in the driving image, and considering the different modes of dilated convolution and pooling operations, two maximum pooling layers of 4×4 and 8×8 are added to enrich the diversity of feature fusion to increase the receptive field; then, an upsampling layer is added to restore the feature size; a 1×1 convolution is added to the first parallel path to retain the original features; in addition, a 1×1 convolution is used in all parallel paths to reduce the number of feature channels to 128; finally, the output features of all parallel paths in the improved ASPP layer are connected in series, and the number of output channels is 768; Step 202: Lightweight OpenPose is used in the key point branch of the model to locate the driver's head and hands, and then RoI Align is used to extract key point features from the global features; Step 203: Considering the differences in driver behavior categories, a key point condition calculation module is proposed to use key point features to conditionally adjust the fully connected layer; the input f of this module k are the key point features extracted from the input driving image, and the expert weights associated with the input The calculation is as follows: r(f k )=S(FC(GAP(f k ))) (1) Where S represents the Sigmoid activation function, FC represents the fully connected layer, and GAP represents the global average pooling layer; the fully connected layer maps the key point features indicating the driving action category to n expert weights; the original parameters of the fully connected layer are set to {W1, W2, …, W n }; W i As the weight of the i-th category feature, that is, the importance of each dimension feature and its impact on the final score; the score of each category is obtained by weighted summation of the features, and then mapped to probability through Softmax; The key point related parameters in the fully connected layer are expressed as {α1·W1, α2·W2, …, α n ·W n }; The final classification feature is calculated as F cls =(α1·W1+…+α n ·W n )*F (2) Where n is the number of experts, · is the dot product operation, and * is the matrix product operation; each kernel W i The dimension of is still the same as that of the original fully connected layer; The weights of the n experts in the fully connected layer depend on the key point features of the driver in the image; these key point features reflect the driving behavior category information and guide different experts to focus on the inputs they are interested in; that is, the weights of the n experts are different for all inputs, where each input can be processed with a different weight; the key point condition calculation module is used to refine the granularity of the condition calculation to the level; the parameters of the fully connected layer used for each driving image in actual prediction are different; Step 3: Train the key point conditional network; based on the deep learning framework Tensorflow, use softmax as the classification loss function; Step 4: Test the keypoint conditioned network.

Images