A method for identifying subway driver behavior

Abstract

The present invention relates to a method for identifying subway driver behavior, including: obtaining historical data of subway driver driving behavior, where the historical data includes various picture data corresponding to different driving behaviors; preprocessing the obtained historical data; dividing the preprocessed historical data into a training set and a test set; constructing an improved ResNet18 model, and iteratively training the improved ResNet18 model using the training set and the test set to obtain a driving behavior recognition model; obtaining actual driving behavior data to be recognized, inputting the driving behavior data into the recognition model, and outputting the corresponding behavior recognition result. Compared with the prior art, the present invention can quickly and accurately classify and identify the driver's driving behavior by designing a lightweight convolutional neural network structure.

Description

Technical Field

[0001] The present invention relates to the technical field of rail transit control, and particularly to a method for identifying subway driver behavior. Background Art

[0002] The main cause of subway accidents is that subway drivers are distracted and inattentive. Among them, the driving behaviors of distracted drivers include using mobile phones while driving, not looking ahead, dozing off, and eating. Therefore, for the current in-vehicle assisted driving system, how to use the existing driver behavior data set to learn the characteristics of various driving behaviors, so as to accurately and efficiently classify and identify driver behaviors and monitor the driver's state will become an effective means to reduce accidents caused by distracted driving.

[0003] The existing in-vehicle assisted driving system generally directly uses convolutional neural networks such as ResNet, InceptionV3, MobileNetV2, and ShuffleNet to extract features and classify according to the obtained category data of driver behaviors to obtain a trained model; then, the trained model is used to input the driver behavior pictures to be classified into the convolutional neural network to obtain the driver behavior classification prediction result. In the selection of convolutional neural networks, networks such as ResNet and InceptionV3 are characterized by high accuracy but complex structures and large model volumes; networks such as MobileNetV2 and ShuffleNet can reduce the amount of computation, but they will cause losses in accuracy. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above-mentioned defects existing in the prior art and provide a method for identifying subway driver behavior, which can quickly and accurately classify and identify driver driving behaviors by designing a lightweight convolutional neural network structure.

[0005] The purpose of the present invention can be achieved by the following technical solutions: A method for identifying subway driver behavior, comprising the following steps:

[0006] S1. Obtain the historical data of subway driver driving behavior, and the historical data includes various picture data corresponding to different driving behaviors;

[0007] S2. Preprocess the obtained historical data;

[0008] S3. Divide the preprocessed historical data into a training set and a test set;

[0009] S4. Construct an improved ResNet18 model, and use the training set and the test set to iteratively train the improved ResNet18 model to obtain a driving behavior recognition model;

[0010] S5. Obtain the actual driving behavior data to be recognized, input the driving behavior data into the recognition model, and output the corresponding behavior recognition result.

[0011] Further, the step S1 is specifically to collect different driving behavior data of subway drivers in the laboratory environment through a camera.

[0012] Further, the different driving behavior data includes 9 types of driving behaviors corresponding to normal, distracted, or dangerous driving states. Specifically, the driving behaviors corresponding to the normal driving state are: sitting normally with both hands on the driving console; sitting normally and making driving gestures.

[0013] The driving behaviors corresponding to the distracted driving state are: drinking water, eating; looking around; playing with mobile phones.

[0014] The driving behaviors corresponding to the dangerous driving state are: the driver lying on the driving console; yawning, the body tilting, the head drooping, or the hand supporting the head; turning around to get something; leaving the driving position.

[0015] Further, the step S2 is specifically to perform cutting, random flipping, shuffling, and linear normalization processing on the obtained historical data.

[0016] Further, the step S3 is specifically to divide the preprocessed historical data into a training set and a test set according to a set ratio.

[0017] Further, the iterative training process of the improved ResNet18 model is: input the data into the model for feature extraction, update the weights through calculating the cross-entropy loss function for backpropagation until the set number of iterations is reached.

[0018] Further, the improved ResNet18 model includes a first convolutional layer, a second module layer, a third module layer, a fourth module layer, a fifth module layer, and an output layer connected in sequence. SE layers are respectively connected between the second module layer and the third module layer, between the third module layer and the fourth module layer, between the fourth module layer and the fifth module layer, and between the fifth module layer and the output layer.

[0019] Further, the first convolutional layer includes a convolutional network, a normalization network, and an activation network connected in sequence. The normalization network uses the BatchNorm2d() function to prevent the network performance from being unstable due to excessive data before activation; the activation network uses the Relu activation function.

[0020] Furthermore, the second module layer, the third module layer, the fourth module layer, and the fifth module layer are all provided with Ghost modules and Involution modules. The Ghost module is used for model compression to improve the calculation speed and reduce the latency.

[0021] The Involution module adopts the neural network operator Involution, and its specific data processing process is as follows:

[0022] 1. Extract the pixels at a certain position on the input channel;

[0023] 2. The φ(X) function generates a kernel of size 1*1*K*K*G and expands it into K×K×G;

[0024] 3. Perform a multiplication and summation operation with the input features to obtain the result 1×1×C.

[0025] Furthermore, the working process of the SE layer includes: First, perform the Squeeze operation, compress the features along the spatial dimension, turn each two-dimensional feature channel into a real number. This real number has a global receptive field, and the output dimension matches the number of input feature channels. It represents the global distribution of responses on the feature channels and enables the layers closer to the input to also obtain a global receptive field.

[0026] Secondly, perform the Excitation operation, which is a mechanism similar to the gate in the recurrent neural network. Generate weights for each feature channel through the parameter w, where the parameter w is learned to explicitly model the correlation between feature channels.

[0027] Finally, perform the Reweight operation, take the weights output by the Excitation as the importance of each feature channel after feature selection, and then weight each channel of the previous feature by multiplication to complete the recalibration of the original features in the channel dimension.

[0028] Compared with the prior art, the present invention proposes a subway driver behavior recognition method based on a lightweight improved ResNet18 model, which adopts the Involution operator, the Ghost module and the SE attention mechanism, and combines with the residual neural network model to design a lightweight convolutional neural network structure. Combining the historical data set collected from the subway simulation cab, the Ghost module is used for model compression, and simple linear operations are used to generate ghosts, which can reduce network parameters and computational complexity while ensuring network accuracy, thereby improving the calculation speed and reducing latency. The Involution operator is used, which is lighter and more efficient than the traditional convolution, and is more concise in form than the traditional self-attention, and can achieve a double improvement in accuracy and efficiency in the models of various visual tasks. The SE layer is used to connect each module layer, the module layer and the output layer to effectively determine the importance corresponding to the extracted features. Thus, a driver behavior recognition model with a small volume, simple operation and high accuracy is trained, which can be well applied to the subway vehicle-mounted mobile terminal scenario for driving behavior recognition and classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic flowchart of the method of the present invention;

[0030] Figure 2 It is a schematic structural diagram of the improved ResNet18 model;

[0031] Figure 3 It is a schematic structural diagram of the convolutional layer in the improved ResNet18 model;

[0032] Figure 4 It is a schematic diagram of the working process of the Ghost module in the module layer;

[0033] Figure 5 It is a schematic diagram of the working process of the Involution module in the module layer;

[0034] Figure 6 It is a schematic diagram of the working process of the SE layer. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Embodiment

[0037] As Figure 1 shown, a subway driver behavior recognition method includes the following steps:

[0038] S1. Obtain the historical data of subway drivers' driving behaviors. The historical data includes various picture data corresponding to different driving behaviors. Specifically, different driving behavior data of subway drivers in a laboratory environment are collected through cameras. Among them, the different driving behavior data includes 9 types of driving behaviors corresponding to normal, distracted, or dangerous driving states. The driving behaviors corresponding to the normal driving state are: sitting normally with both hands on the driving console; sitting normally and making driving gestures;

[0039] The driving behaviors corresponding to the distracted driving state are: drinking water, eating; looking around; playing with mobile phones;

[0040] The driving behaviors corresponding to the dangerous driving state are: the driver lying on the driving console; yawning, body tilting, head lowering, or hands supporting the head; turning around to get something; leaving the driving position;

[0041] S2. Preprocess the obtained historical data. Specifically, perform cutting, random flipping, shuffling, and linear normalization on the obtained historical data;

[0042] S3. Divide the preprocessed historical data into a training set and a test set according to a set ratio;

[0043] S4. Construct an improved ResNet18 model, and use the training set and the test set to iteratively train the improved ResNet18 model to obtain a driving behavior recognition model. Among them, the iterative training process of the improved ResNet18 model is: input the data into the model for feature extraction, calculate the cross-entropy loss function for backpropagation to update the weights until the set number of iterations is reached;

[0044] S5. Obtain the actual driving behavior data to be recognized, input the driving behavior data into the recognition model, and output the corresponding behavior recognition result.

[0045] As Figure 2 shown, in this technical solution, the improved ResNet18 model includes a first convolutional layer, a second module layer, a third module layer, a fourth module layer, a fifth module layer, and an output layer connected in sequence. An SE layer is respectively connected between the second module layer and the third module layer, between the third module layer and the fourth module layer, between the fourth module layer and the fifth module layer, and between the fifth module layer and the output layer;

[0046] As Figure 3 shown, the first convolutional layer includes a convolutional network, a normalization network, and an activation network connected in sequence. The normalization network uses the BatchNorm2d() function to prevent the network performance from being unstable due to excessive data before activation; the activation network uses the Relu activation function;

[0047] The second module layer, the third module layer, the fourth module layer, and the fifth module layer are all provided with a Ghost module and an Involution module. The Ghost module is used for model compression to improve the calculation speed and reduce the latency.

[0048] The Involution module adopts the neural network operator Involution, and its specific data processing process is as follows:

[0049] 1. Extract the pixel at a certain position on the input channel;

[0050] 2. The φ(X) function generates a kernel of size 1*1*K*K*G and expands it into K×K×G;

[0051] 3. Perform multiplication and summation operations with the input features to obtain the result 1×1×C;

[0052] The working process of the SE layer includes: First, perform the Squeeze operation, compress the features along the spatial dimension, turn each two-dimensional feature channel into a real number. This real number has a global receptive field, and the output dimension matches the number of input feature channels. It represents the global distribution of responses on the feature channels and enables the layers close to the input to also obtain a global receptive field;

[0053] Secondly, perform the Excitation operation, which is a mechanism similar to the gate in the recurrent neural network. Generate weights for each feature channel through the parameter w. Among them, the parameter w is learned to explicitly model the correlation between feature channels;

[0054] Finally, perform the Reweight operation, take the weights output by the Excitation as the importance of each feature channel after feature selection, and then weight each channel of the previous feature by multiplication to complete the recalibration of the original features in the channel dimension.

[0055] This embodiment applies the above technical solutions, mainly including the following contents:

[0056] First, obtain various behavior data collected in the laboratory and get a series of photos corresponding to different driving behaviors;

[0057] After that, preprocess the pictures, shuffle the data, and perform operations such as stretching and flipping on the pictures;

[0058] Then divide the data into a training set and a test set;

[0059] Then input the data into the improved ResNet18 model built for feature extraction;

[0060] Update the weights by backpropagation through calculating the cross-entropy loss function;

[0061] And through multiple rounds of iteration, a relatively optimal training model is obtained;

[0062] Finally, the trained model is deployed to the required scenarios to classify various behaviors of subway drivers.

[0063] In the constructed model, its main structures are:

[0064] Involution operator, the involution kernel is generated by a function conditional on a single pixel at (i,j), followed by channel-to-space rearrangement; subsequently, the multiplication-addition operation of involution is decomposed into 2 steps, representing the multiplication operation propagating across C channels, representing the summation operation aggregating within the spatial neighborhood;

[0065] The Ghost module is a method of model compression that generates ghosts using simple linear operations, i.e., reducing network parameters and computational complexity while ensuring network accuracy, thereby improving computational speed and reducing latency;

[0066] SE structure: A SE module process is mainly divided into two steps: Squeeze and Excitation. Squeeze obtains the global compressed feature quantity of the current Feature Map by performing Global Average Pooling on the Feature Map layer; Excitation obtains the weights of each channel in the Feature Map through a two-layer fully connected bottleneck structure and uses the weighted Feature Map as the input of the next layer of the network.

[0067] In this embodiment, the data collected in the laboratory includes 9 categories of driver behavior data, which belong to three states: normal, distracted, or dangerous driving.

[0068] Normal driving: 1) Normal sitting state, hands on the driver's console; 2) Normal sitting state, making gestures (left hand indicating forward or raising the left hand in a fist).

[0069] Distracted driving: 1) Drinking water, eating; 2) Looking around; 3) Playing with mobile phones.

[0070] Dangerous driving: 1) The driver lying on the driver's console; 2) Yawning, body tilting, or head lowering, hand supporting the head; 3) Reaching behind to get something; 4) Leaving the driver's seat.

[0071] After obtaining the 9 categories of behavior picture data of subway drivers in the laboratory environment, they are saved according to the behavior labels as file names and read in sequentially;

[0072] The data preprocessing process is as follows: First, the pictures are cut into a unified size of 224 * 224 pixels, then randomly flipped and shuffled and linearly normalized, and then divided into a training set and a test set in a ratio of 8:2.

[0073] The input data is a three-channel color image of 224 * 224, which is input as the input end into the improved ResNet18 model for feature extraction.

[0074] The improved ResNet18 is improved on the basis of the original ResNet18 model (as shown in Table 1: including a convolutional layer, 4 modules and an output layer). The improved model is shown in Table 2. In each module, it is replaced with the Ghost ghost processing method in GhostNet, and the 7 * 7 Involution convolution operator replaces the original 3 * 3 convolution method. The output layer first performs max pooling processing and then outputs the probability values of various behavior categories through a softmax classifier.

[0075] Table 1

[0076]

[0077]

[0078] Table 2

[0079]

[0080] The convolutional layer includes a convolution, normalization, and activation function connected in sequence. In this embodiment, the normalization process uses the BatchNorm2d() function, so that the data will not cause instability of the network performance due to excessive data before the activation function. The mathematical principle of the BatchNorm2d() function is as follows:

[0081]

[0082] Among them, mean is the mean of the data, num_features is the total number of features. Generally, the input parameters include batch_size * num_features * height * width, eps is a value added to the denominator for the purpose of calculation stability, and the default value is 1e-5. momentum is an estimation parameter for the mean and variance during the running process. affine: When set to true, it will give the learnable coefficient matrices gamma and beta;

[0083] The Relu function performs a non-linear transformation. The formula of the Relu activation function is as follows:

[0084]

[0085] The Ghost module adopts the Ghost module in the GhostNet network. The Ghost module is a model compression method, which can reduce network parameters and computational complexity while ensuring network accuracy, thereby improving the computational speed (speed) and reducing the latency. As Figure 4 shown, the principle is as follows: The process of generating the ghost map uses a simple linear operation Φ to replace the original convolution operation. Suppose that in the original convolution operation, the input Input is convolved with n groups of k x k Kernels to generate an output with n channels and a size of h’×w’. In the ghost model, m groups of k x k Kernels are used to convolve with the input to generate an intrinsic map of m×h’×w’. Then, the intrinsic map undergoes a linear transformation Φ to generate the ghost map, and both the intrinsic and the ghost are used as the output.

[0086] Involution module: As Figure 5 shown, the core idea is the opposite of convolution. It reconsiders the characteristics of convolution and starts from channel redundancy. It is lighter and more efficient than convolution, and is more concise in form than self-attention. It can be used in models for various vision tasks to achieve a double improvement in accuracy and efficiency.

[0087] The steps are as follows: Extract the pixel at a certain position on the input channel;

[0088] The φ(X) function generates a kernel of size 1*1*K*K*G and expands it to K×K×G;

[0089] Perform a multiplication and summation operation with the input feature to obtain the result 1×1×C.

[0090] The SE module is designed considering the relationship between feature channels. Its principle is as Figure 6 shown. Given an input x with c1 feature channels, after a series of general transformations such as convolution, a feature with c2 feature channels is obtained.

[0091] First is the Squeeze operation, which compresses the features along the spatial dimension, turning each two-dimensional feature channel into a real number. This real number has a global receptive field to some extent, and the output dimension matches the number of input feature channels. It represents the global distribution of responses on the feature channels and enables layers closer to the input to also obtain a global receptive field, which is very useful in many tasks.

[0092] Next is the Excitation operation, which is a mechanism similar to the gates in recurrent neural networks. Weights are generated for each feature channel through the parameter w, where the parameter w is learned to explicitly model the correlations between feature channels.

[0093] Finally, there is a Reweight operation. The weights of the output of Excitation are regarded as the importance of each feature channel after feature selection, and then the original features are re-calibrated in the channel dimension by multiplying and weighting each channel to the previous features.

[0094] The classifier in the output layer uses a softmax classifier for multi-classification:

[0095]

[0096] Among them, q is the number of finally output categories. In the action classification task, the number of categories is 9. It can be seen from the above formula that the sum of the predicted values of each sample for 9 categories (numbers 0-8) is equal to 1.

[0097] In summary, in this technical solution, the constructed improved model is lightweight, and both the model parameters and the computational amount are greatly reduced compared with the original residual neural network.

[0098] The improved model uses the Involution operator to replace the original convolutional block and adds an attention mechanism, which not only optimizes the overfitting problem but also improves the model accuracy.

[0099] The source code of the improved model can adopt the Pytorch architecture, which is convenient for later modification and deployment and has strong portability.

Claims

1. A method for identifying subway driver behavior, characterized in that, It includes the following steps: S1. Obtain the historical data of the subway driver's driving behavior, where the historical data contains various picture data corresponding to different driving behaviors; S2. Preprocess the obtained historical data; S3. Divide the preprocessed historical data into a training set and a test set; S4. Construct an improved ResNet18 model, and use the training set and the test set to iteratively train the improved ResNet18 model to obtain a driving behavior recognition model; The improved ResNet18 model includes a first convolutional layer, a second module layer, a third module layer, a fourth module layer, a fifth module layer, and an output layer connected in sequence. An SE layer is connected between the second module layer and the third module layer, between the third module layer and the fourth module layer, between the fourth module layer and the fifth module layer, and between the fifth module layer and the output layer respectively; The first convolutional layer includes a convolutional network, a normalization network, and an activation network connected in sequence. The normalization network uses the BatchNorm2d() function to prevent the network performance from being unstable due to excessive data before activation; the activation network uses the Relu activation function; The second module layer, the third module layer, the fourth module layer, and the fifth module layer are all provided with a Ghost module and an Involution module. The Ghost module is used for model compression to improve the calculation speed and reduce the latency; The Involution module uses the neural network operator Involution, and its specific data processing process is as follows:

1. Extract the pixel at a certain position on the input channel; 2、 The function generates a kernel of size 1*1*K*K*G and expands it to K×K×G; 3. Perform a multiplication and addition operation with the input feature to obtain a result of 1×1×C; S5. Obtain the actual driving behavior data to be recognized, input the driving behavior data into the recognition model, and output the corresponding behavior recognition result.

2. The subway driver behavior recognition method according to claim 1, characterized in that The step S1 is specifically to collect different driving behavior data of subway drivers in the laboratory environment through a camera.

3. A subway driver behavior recognition method according to any one of claims 1 to 2, characterized in that, The different driving behavior data includes 9 types of driving behaviors corresponding to normal, distracted, or dangerous driving states. Specifically, the driving behaviors corresponding to the normal driving state are: sitting normally with both hands on the driving console; sitting normally and making driving gestures; The driving behaviors corresponding to the distracted driving state are: drinking water, eating; looking around; playing with mobile phones; The driving behaviors corresponding to the dangerous driving state are: the driver lying on the driving console; yawning, the body tilting, the head drooping, or the hand supporting the head; turning around to get something; leaving the driving position.

4. The subway driver behavior recognition method according to claim 1, wherein The step S2 is specifically to perform cutting, random flipping, shuffling, and linear normalization processing on the obtained historical data.

5. A subway driver behavior recognition method according to claim 1, characterized in that, The step S3 is specifically to divide the preprocessed historical data into a training set and a test set according to a set ratio.

6. The subway driver behavior recognition method according to claim 1, characterized in that, The iterative training process of the improved ResNet18 model is: input the data into the model for feature extraction, and update the weights by calculating the cross-entropy loss function for backpropagation until the set number of iterations is reached.

7. A subway driver behavior recognition method according to claim 1, characterized in that The working process of the SE layer includes: First, perform the Squeeze operation, which compresses features along the spatial dimension. Each two-dimensional feature channel is transformed into a real number. This real number has a global receptive field, and the output dimension matches the number of input feature channels. It represents the global distribution of responses on the feature channels and enables layers closer to the input to also obtain a global receptive field. Second, perform the Excitation operation, which is a mechanism similar to the gate in a recurrent neural network. Weights are generated for each feature channel through the parameter w. Among them, the parameter w is learned to explicitly model the correlation between feature channels. Finally, perform the Reweight operation. The weights output by Excitation are used as the importance of each feature channel after feature selection, and then the original features are weighted channel by channel through multiplication to complete the recalibration of the original features in the channel dimension.

Images