A safe driving monitoring method using feature adaptive weighting

Abstract

The present invention discloses a safe driving monitoring method using feature adaptive weighting. The present invention studies the fusion strategy of global features and key point features of different scales. In view of the attention problem in the fusion process, the present invention does not directly cascade global features and key point features, but proposes a posture-based feature fusion module for both. The category differences of driver behavior are manifested in different image regions. The model should focus on different regions of different input images. Therefore, an adaptive weighting module is proposed. By learning a set of expert weights specific to the input data to select the convolution kernel for calculation, it provides a new direction for driver action recognition and further improves the accuracy of driver behavior recognition. The present invention has important application value in the field of traffic safety.

Description

Technical Field

[0001] The invention belongs to the field of image processing and pattern recognition, and in particular relates to a safe driving monitoring method using feature adaptive weighting. Background Art

[0002] Despite improvements in the safety of road and vehicle designs, the total number of fatal accidents is still increasing. The World Health Organization (WHO) 2017 Global Status Report reported that an estimated 1.25 million people die each year from road traffic accidents worldwide, and road traffic accidents cause up to 50 million non-fatal injuries. In addition, road traffic accidents cause huge property losses, and the number of road accidents caused by distracted driving is steadily increasing, so the research on driver behavior recognition algorithms is an important but challenging task for road safety. Summary of the invention

[0003] To solve the above problems, the present invention discloses a safe driving monitoring method using feature adaptive weighting. The feature fusion and dynamic convolution methods used in the invention improve the accuracy of driver behavior recognition in the testing phase.

[0004] To achieve the above object, the technical solution of the present invention is as follows:

[0005] A safe driving monitoring method using feature adaptive weighting comprises the following steps:

[0006] Step 1: Use the existing StateFarm distracted driving dataset as the experimental dataset;

[0007] Step 2: Construct a feature adaptive weighted model. The present invention uses ResNet as a network global feature extractor, uses a posture estimation model to capture key point level semantics, and fuses global features and key point features through a posture-based feature fusion module. The module adopts a multi-branch structure, one branch uses a global average pooling layer to extract the attention of global features, and the other branch directly uses point-by-point convolution to extract the channel attention of key point features; then, the fused features are fed into an adaptive weighting module to dynamically adjust the convolutional neural network; a set of input-dependent weights are dynamically generated using a fully connected layer, and combined with corresponding convolution parameters to generate a new convolution kernel; finally, the output features of the module are input into a classifier;

[0008] Step 201: For the input driving image, ResNet is used as the model backbone to extract global features;

[0009] Step 202: Use the posture estimation model to detect the key points of the driver, generate the bounding box of the key points through post-processing, and then extract the key point features and model them through RoI Align; since there are cases where the key points cannot be detected due to occlusion and other reasons, a threshold is set for the key point response; key points with response values ​​below the threshold will not participate in subsequent calculations; the above network generates a separate feature map for each key point;

[0010] Step 203: The category differences of the driver's actions are mainly reflected in the key point details, so the global features and key point features are cascaded and fused;

[0011] Step 204: In order to enhance the representation of useful key point feature channels and suppress irrelevant features, an adaptive weighting module is proposed to recalibrate the activation strength of different key point feature channels; the adaptive weighting module is used to transform the fused features before passing them to the classifier; each input is taken as a linear combination of n experts to calculate the convolution kernel. The following is a detailed description of the module:

[0012] The adaptive weighting module sets multiple convolution kernels in the convolution layer; the weight of each convolution kernel is determined by the input of the convolution layer through a fully connected layer; finally, a set of convolution kernels tailored for the input is obtained by weighted summation, thereby realizing a convolution; the fusion features of the global and key points are used as the input of the adaptive weighting module; the fusion features reveal the driver's action category and guide different experts to focus on the input they are interested in; the n expert weights in the convolution layer are determined by the fusion features; in other words, the weights of the n experts are different in all samples, and each input is processed with a different weight; specifically, the expert weight α = r (f) is dynamically generated and combined with the corresponding original parameters to generate a new convolution kernel;

[0013] α=r(f)=S(FC(GAP(f)))#(5)

[0014] Where S represents the Sigmoid activation function, FC represents the fully connected layer, and GAP represents the global average pooling layer; the fully connected layer maps the processed fusion features to n expert weights; the convolution kernel related to the input is calculated according to the function of the input sample and parameterized as (α1·W1+…α n ·W n );Generated convolution feature f ID The calculation is as follows:

[0015] f ID =σ((α1·W1+…a n ·W n )*f)#(6)

[0016] Where f represents the input feature, each α i =ri (x) is a scalar weight that depends on the input, n is the number of experts, and σ is an activation function. It is obvious that for different inputs, the model capacity increases with the number of experts. It only requires a small inference cost because the convolution kernel is calculated as the linear sum of the n expert weights instead of increasing the kernel parameters or the number of channels of the convolution layer.

[0017] We use adaptive weighting modules to replace standard convolutional layers to build adaptive blocks, which can learn specific convolution kernel parameters for each input driving image. To avoid serious overfitting problems caused by too deep a network, and to make the expert weights in the adaptive weighting module more class-specific at deeper layers of the network, we use adaptive blocks only in the last convolution group in ResNet.

[0018] Step 3: Train the feature adaptive weighting model using the SGD optimizer on the open source platform PyTorch.

[0019] Step 4: Test the feature adaptive weighting model.

[0020] The beneficial effects of the present invention are:

[0021] (1) The present invention proposes a posture-based feature fusion module to fuse global and key point features, which enhances channel attention and fuses multi-scale feature contexts.

[0022] (2) The present invention proposes an adaptive weighting module, which customizes a separate convolution layer for each input sample and performs dynamic adjustment. This module improves the feature representation capability without increasing the amount of computation.

[0023] (3) The present invention further improves the accuracy of driver behavior recognition and has important application value in the field of traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the present invention;

[0025] Figure 2 are sample pictures of different driving behaviors in the present invention,

[0026] Figure 3 is a schematic diagram of the structure of the feature adaptive weighted model in the present invention,

[0027] Figure 4 It is a structural diagram of the adaptive weighting module in the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0029] The safe driving monitoring method using feature adaptive weighting described in the present invention has the following specific implementation steps:

[0030] Step 1: Use the existing StateFarm distracted driving dataset as the experimental dataset;

[0031] Step 2: Build a feature adaptive weighted model, Figure 3 is a schematic diagram of the structure of the model; the present invention uses ResNet as a network global feature extractor, uses a posture estimation model to capture key point level semantics, and fuses global features and key point features through a posture-based feature fusion module. The module adopts a multi-branch structure, one branch uses a global average pooling layer to extract the attention of global features, and the other branch directly uses point-by-point convolution to extract the channel attention of key point features; then, the fused features are fed into an adaptive weighting module to dynamically adjust the convolutional neural network; a set of input-dependent weights are dynamically generated using a fully connected layer, and combined with corresponding convolution parameters to generate a new convolution kernel; finally, the output features of the module are input into a classifier;

[0032] Step 201: For the input driving image, ResNet is used as the model backbone to extract global features;

[0033] Step 202: Use the posture estimation model to detect the key points of the driver, generate the bounding box of the key points through post-processing, and then extract the key point features and model them through RoI Align; since there are cases where the key points cannot be detected due to occlusion and other reasons, a threshold is set for the key point response; key points with response values ​​below the threshold will not participate in subsequent calculations; the above network generates a separate feature map for each key point;

[0034] Step 203: The category differences of the driver's actions are mainly reflected in the key point details, so the global features and key point features are cascaded and fused;

[0035] Step 204: In order to enhance the representation of useful key point feature channels and suppress irrelevant features, an adaptive weighting module is proposed to recalibrate the activation strength of different key point feature channels; Figure 3 In , the adaptive weighting module is used to transform the fused features before passing them to the classifier; Figure 4 As shown in Figure 1, each input is taken as a linear combination of n experts to calculate the convolution kernel. The following is a detailed description of this module:

[0036] The adaptive weighting module sets multiple convolution kernels in the convolution layer; the weight of each convolution kernel is determined by the input of the convolution layer through a fully connected layer; finally, a set of convolution kernels tailored for the input is obtained by weighted summation, thereby realizing a convolution; the fusion features of the global and key points are used as the input of the adaptive weighting module; the fusion features reveal the driver's action category and guide different experts to focus on the input they are interested in; the weights of the n experts in the convolution layer are determined by the fusion features; in other words, the weights of the n experts are different in all samples, and each input is processed with a different weight; specifically, the expert weight a=r(f) is dynamically generated and combined with the corresponding original parameters to generate a new convolution kernel;

[0037] α=r(f)=S(FC(GAP(f)))#(5)

[0038] Where S represents the Sigmoid activation function, FC represents the fully connected layer, and GAP represents the global average pooling layer; the fully connected layer maps the processed fusion features to n expert weights; the convolution kernel related to the input is calculated according to the function of the input sample and parameterized as (α1·W1+…α n ·W n );Generated convolution feature f ID The calculation is as follows:

[0039] f ID =σ((α1·W1+…α n ·W n )*f)#(6)

[0040] Where f represents the input feature, each α i =r i (x) is a scalar weight that depends on the input, n is the number of experts, and σ is an activation function. It is obvious that for different inputs, the model capacity increases with the number of experts. It only requires a small inference cost because the convolution kernel is calculated as the linear sum of the n expert weights instead of increasing the kernel parameters or the number of channels of the convolution layer.

[0041] We use adaptive weighting modules to replace standard convolutional layers to build adaptive blocks, which can learn specific convolution kernel parameters for each input driving image. To avoid serious overfitting problems caused by too deep a network, and to make the expert weights in the adaptive weighting module more class-specific at deeper layers of the network, we use adaptive blocks only in the last convolution group in ResNet.

[0042] Step 3: Train the feature adaptive weighting model using the SGD optimizer on the open source platform PyTorch.

[0043] Step 4: Test the feature adaptive weighting model.

[0044] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A safe driving monitoring method using feature adaptive weighting, characterized in that: The following steps are involved: Step 1: Use the existing StateFarm distracted driving dataset as the experimental dataset; Step 2: Construct a feature adaptive weighted model; ResNet is used as the network global feature extractor, and the posture estimation model is used to capture the key point level semantics. The global features and key point features are fused through the posture-based feature fusion module. The module adopts a multi-branch structure. One branch uses the global average pooling layer to extract the attention of the global features, and the other branch directly uses point-by-point convolution to extract the channel attention of the key point features. Subsequently, the fused features are fed into the adaptive weighting module to dynamically adjust the convolutional neural network. A set of input-dependent weights are dynamically generated using the fully connected layer and combined with the corresponding convolution parameters to generate a new convolution kernel. Finally, the output features of the module are input to the classifier. The specific steps are as follows: Step 201: For the input driving image, ResNet is used as the model backbone to extract global features; Step 202: Use the posture estimation model to detect the key points of the driver, generate the bounding box of the key points through post-processing, and then extract the key point features and model them through RoI Align; since there is a situation where the key points cannot be detected due to occlusion, a threshold is set for the key point response; the key points with response values ​​below the threshold will not participate in subsequent calculations; the above network generates a separate feature map for each key point; Step 203: The category differences of the driver's actions are reflected in the key point details, so the global features and key point features are cascaded and fused; Step 204: In order to enhance the representation of useful key point feature channels and suppress irrelevant features, an adaptive weighting module is proposed to recalibrate the activation strength of different key point feature channels; the adaptive weighting module is used to transform the fused features before passing them to the classifier; each input is taken as a linear combination of n experts to calculate the convolution kernel. The following is a detailed description of the module: The adaptive weighting module sets multiple convolution kernels in the convolution layer; the weight of each convolution kernel is determined by the input of the convolution layer through a fully connected layer; finally, a set of convolution kernels tailored for the input is obtained by weighted summation, thereby realizing a convolution; the fusion features of the global and key points are used as the input of the adaptive weighting module; The fused features reveal the driver action category and guide different experts to focus on the inputs they are interested in. The weights of the n experts in the convolutional layer are determined by the fused features. In other words, the weights of the n experts are different in all samples, and each input is processed with a different weight. Specifically, expert weights α = r(f) are dynamically generated and combined with the corresponding original parameters to generate new convolution kernels; α=r(f)=S(FC(GAP(f)))#(5) Where S represents the Sigmoid activation function, FC represents the fully connected layer, and GAP represents the global average pooling layer; the fully connected layer maps the processed fusion features to n expert weights; the convolution kernel related to the input is calculated according to the function of the input sample and parameterized as (α1·W1+…α n ·W n );Generated convolution feature f ID The calculation is as follows: f ID =σ((α1·W1+…α n ·W n )*f)#(6) Where f represents the input feature, each α i =r i (x) is a scalar weight that depends on the input, n is the number of experts, and σ is an activation function. It is obvious that for different inputs, the model capacity increases with the number of experts. It only requires a small inference cost because the convolution kernel is calculated as the linear sum of the n expert weights instead of increasing the kernel parameters or the number of channels of the convolution layer. We use adaptive weighting modules to replace standard convolutional layers to build adaptive blocks, which can learn convolution kernel parameters for each input driving image. To avoid serious overfitting problems caused by too deep a network, and to make the expert weights in the adaptive weighting module more class-specific at deeper layers of the network, we use adaptive blocks only in the last convolution group in ResNet. Step 3: Train the feature adaptive weighting model using the SGD optimizer on the open source platform PyTorch. Step 4: Test the feature adaptive weighting model.

Images