Truck driver posture behavior recognition method based on convolutional neural network fusion model

Abstract

The application provides a truck driver posture behavior recognition method based on a convolutional neural network fusion model, which comprises the following steps: constructing a standard image data set of truck driver posture behaviors; constructing four models for truck driver posture behavior recognition, and obtaining the optimal performance of RetinaNet-DPBR through comparative experiments; using RetinaNet-DPBR to perform driving posture behavior region cutting and extraction on the standard image data set; constructing a truck driver posture behavior recognition model VGG16-DPBR based on a convolutional neural network, obtaining a one-dimensional feature vector F V ; constructing a truck driver posture behavior recognition model Xception-DPBR based on a convolutional neural network, obtaining a one-dimensional feature vector F X ; constructing a truck driver posture behavior recognition model MobileNetV2-DPBR based on a convolutional neural network, obtaining a one-dimensional feature vector F M ; and fusing the feature vectors F V , F X and F M to construct a convolutional neural network fusion model FCNN-DPBR, so that the truck driver posture behavior can be more accurately recognized, and a certain technical guarantee can be provided for the safety of the driver during driving.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of intelligent transportation and automatic driving, and particularly relates to a truck driver posture behavior recognition method based on a convolutional neural network fusion model. BACKGROUND

[0002] In a complex dynamic traffic system, the wrong driving behavior and body posture of a driver are the main factors causing traffic accidents. Therefore, it is particularly important to detect and identify the driving posture behavior of a driver and provide a prompt and a wake-up when the driver has a wrong driving posture behavior. On the one hand, it can not only reduce the probability of road traffic accidents and the severity of traffic accidents, but also protect the safety of citizens in China and create a safe and orderly road driving environment. At the same time, it has great significance for the application and popularization of the field of future vehicle automatic driving.

[0003] In the research of driving posture behavior recognition, traditional machine learning methods need professional ability and a large amount of time to select features and set parameters, have low calculation efficiency, and are difficult to apply to actual traffic scenes. Deep learning does not rely on prior knowledge like machine learning algorithms, only needs to construct a neural network model, and can be applied to the test set for testing while having observable detection accuracy and generalization ability.

[0004] The fusion network model can combine the characteristics of single models, make up for the defects of single models, adapt to more scene conditions, and greatly improve the comprehensive performance of the model. Therefore, the present application provides a truck driver posture behavior recognition method based on a convolutional neural network fusion model. SUMMARY

[0005] The present application aims to overcome the deficiencies in the prior art and provide a truck driver posture behavior recognition method based on a convolutional neural network fusion model. The RetinaNet-DPBR model is used to crop and extract the driving posture behavior region in the truck driver posture behavior standard data set image, and a deep fusion neural network is used to effectively recognize and classify the driving posture behavior, thereby providing a certain technical guarantee for the safety of the driver during driving.

[0006] Technical solution: To achieve the above purpose, the present application provides a truck driver posture behavior recognition method based on a convolutional neural network fusion model, comprising the following steps:

[0007] S1: Construct a truck driver posture behavior standard image data set, including Kaggle open source data set and SEU self-made data set;

[0008] S2: Four models for truck driver posture behavior recognition are constructed: Faster R-CNN-DPBR, SSD-DPBR, YOLOv5-DPBR, and RetinaNet-DPBR. Based on the two standard image datasets constructed and comparative experiments, it is concluded that the model most suitable for truck driver posture behavior recognition task and with the best comprehensive performance is RetinaNet-DPBR;

[0009] S3: The RetinaNet-DPBR model is used to crop and extract the truck driver posture behavior standard image dataset, to obtain the driving posture behavior region in the image, so as to construct the truck driver posture behavior core image dataset, including KaggleCrop dataset and SEUCrop dataset;

[0010] S4: A truck driver posture behavior recognition model VGG16-DPBR based on convolutional neural network is constructed to obtain one-dimensional feature vector F V ;

[0011] S5: A truck driver posture behavior recognition model Xception-DPBR based on convolutional neural network is constructed to obtain one-dimensional feature vector F X ;

[0012] S6: A truck driver posture behavior recognition model MobileNetV2-DPBR based on convolutional neural network is constructed to obtain one-dimensional feature vector F M ;

[0013] S7: The feature vectors FV, FX, and FM are fused based on the parallel rule to construct a convolutional neural network fusion model FCNN-DPBR, and the truck driver posture behavior is recognized.

[0014] Further, the specific method for constructing the truck driver posture behavior standard image dataset in step S1 is as follows:

[0015] S1-1: For the Kaggle truck driver posture behavior standard dataset, 10 driving posture behavior categories including safe driving, left hand sending information, left hand making a phone call, right hand sending information, right hand making a phone call, operating the center console, drinking water, body stretching back, arranging hair or makeup, and talking with passengers are included. 1500 images of each category are randomly extracted from the original Kaggle dataset, covering male and female drivers of different ages from 20 to 60 years old, and skin colors including yellow, white, and black.

[0016] S1-2: For the SEU truck driver posture behavior standard data set, it contains four driving posture behavior categories: normal driving, steering displacement, talking on the phone and eating. Through data enhancement, the number of images in each category is increased to 400, covering 11 male drivers of different ages from 20 to 60 years old;

[0017] S1-3: Define the driving posture behavior region in the data set image. The purpose is to use as little image information as possible to describe the truck driver posture behavior, and to lay a good foundation for subsequent data labeling and target detection positioning;

[0018] S1-4: Use LabelMe software to label the driving posture behavior region in the two truck driver posture behavior standard data set images. The label name of the Kaggle data set is: {c0, c1, c2, c3, c4, c5, c6, c7, c8, c9}, and the label name of the SEU data set is: {Pose1, Pose2, Pose3, Pose4};

[0019] S1-5: Convert the.json format label file data set generated by using LabelMe software into a standard VOC data set format, which is:

[0020] VOCdevkit

[0021] VOC2007

[0022] Annotations

[0023] ImageSets

[0024] Main

[0025] JPEGImages

[0026] Among them, the Annotations folder stores.xml format label files; The ImageSets folder has a Main folder under it, which contains data set division information; The JPEGImages folder stores original image data;

[0027] S1-6: Before training each model, randomly extract image data and divide it into training set, validation set and test set according to the ratio of 4:1:1.

[0028] Further, the specific steps of constructing four models for truck driver posture behavior recognition in step S2 are as follows:

[0029] S2-1: Construct a Faster R-CNN-DPBR model for truck driver posture behavior perception recognition;

[0030] S2-2: Construct an SSD-DPBR model for truck driver posture behavior perception recognition;

[0031] S2-3: Construct a YOLOv5-DPBR model for truck driver posture behavior perception recognition;

[0032] S2-4: Construct a RetinaNet-DPBR model for truck driver posture behavior perception recognition;

[0033] S2-5: Conduct a comparative experiment to compare the test results of the models on the Kaggle dataset and the SEU dataset, and determine the best model in terms of comprehensive performance in the two datasets.

[0034] Further, the specific content of the comparative experiment in step S2-5 is: ① Select experimental evaluation indicators, use precision, recall and F1-score as accuracy indicators, and average detection time (Times, in ms, and take integer) as real-time indicator; ② In the experimental stage, the corresponding hyperparameters need to be set, and the change of the loss function curve is observed. If the loss function curve gradually decreases and finally remains stable, it indicates that the curve has converged. If it is always in a downward trend, the value of the hyperparameter should be adjusted until the curve converges; ③ Apply the trained model to the test set for detection, and calculate the evaluation indicators of each model. Comprehensive comparison shows that the comprehensive performance of the RetinaNet-DPBR model is better than that of the other three models, indicating that the RetinaNet-DPBR model is more suitable for truck driver posture behavior perception recognition task.

[0035] Further, the specific content of using the RetinaNet-DPBR model to crop and extract the truck driver posture behavior standard image dataset in step S3 is:

[0036] S3-1: Use the RetinaNet-DPBR model to crop and extract the driving posture behavior region in the Kaggle dataset image, thereby constructing the KaggleCrop truck driver posture behavior core dataset;

[0037] S3-2: Use the RetinaNet-DPBR model to crop and extract the driving posture behavior region in the SEU dataset image, thereby constructing the SEUCrop truck driver posture behavior core dataset;

[0038] S3-3: Based on the constructed truck driver posture behavior core image dataset, before training each model, randomly extract image data and divide them into training set, validation set and test set according to the ratio of 4:1:1.

[0039] Further, the step S4 of constructing the truck driver posture behavior recognition model VGG16-DPBR based on the convolutional neural network obtains a one-dimensional feature vector F V The specific method is: based on the original structure of the VGG16 model, remove the top three fully connected layers, and on this basis, add a global average pooling layer, a Dropout layer with a dropout rate of 0.5, and a fully connected layer with a softmax activation function. The network recognition categories are: safe driving, left hand texting, left hand calling, right hand texting, right hand calling, operating center console, drinking, body stretching back, arranging hair or makeup, and talking with passengers in the KaggleCrop dataset, and normal driving, steering displacement, calling, and eating in the SEUCrop dataset. Thus, the truck driver posture behavior recognition model VGG16-DPBR based on the convolutional neural network is obtained. After training, the top fully connected layer and the Dropout layer are deleted, and finally a one-dimensional feature vector F V of 1×1×512 is output.

[0040] Further, the step S5 of constructing the truck driver posture behavior recognition model Xception-DPBR based on the convolutional neural network obtains a one-dimensional feature vector F X The specific method is: based on the original structure of the Xception model, remove the top three fully connected layers, and on this basis, add a global average pooling layer, a Dropout layer with a dropout rate of 0.5, and a fully connected layer with a softmax activation function. The network recognition categories are: safe driving, left hand texting, left hand calling, right hand texting, right hand calling, operating center console, drinking, body stretching back, arranging hair or makeup, and talking with passengers in the KaggleCrop dataset, and normal driving, steering displacement, calling, and eating in the SEUCrop dataset. Thus, the truck driver posture behavior recognition model Xception-DPBR based on the convolutional neural network is obtained. After training, the top fully connected layer and the Dropout layer are deleted, and finally a one-dimensional feature vector F X of 1×1×2048 is output.

[0041] Further, the step S6 of constructing the truck driver posture behavior recognition model MobileNetV2-DPBR based on the convolutional neural network obtains a one-dimensional feature vector F MThe specific method is: based on the original structure of the MobileNetV2 model, removing the top three fully connected layers, and adding a global average pooling layer, a Dropout layer with a dropout rate of 0.5 and a fully connected layer with a softmax activation function, the network identifies 10 categories: safe driving, left hand texting, left hand calling, right hand texting, right hand calling, operating center console, drinking, stretching back, arranging hair or makeup, and talking with passengers in the KaggleCrop dataset, and 4 categories: normal driving, steering, calling and eating in the SEUCrop dataset, thereby obtaining a truck driver posture behavior recognition model MobileNetV2-DPBR based on a convolutional neural network, after training, the top fully connected layer and the Dropout layer are deleted, and finally a one-dimensional feature vector F of 1*1*1280 is output M .

[0042] Further, the feature vectors F V , F X , F M are fused based on the parallel rule, thereby constructing a convolutional neural network fusion model FCNN-DPBR. V , F X , F M The one-dimensional feature vector of 1*1*3840 is obtained based on the parallel rule fusion, then the merged one-dimensional feature vector is input, a Dropout layer is added again to prevent overfitting, and a fully connected layer with a softmax activation function outputs a prediction result, the fusion model fully connected layer is retrained, and finally the fusion model FCNN-DPBR is obtained.

[0043] Advantages: compared with the prior art, the automatic recognition of the truck driver posture behavior is realized based on the convolutional neural network technology, the recognition accuracy, model stability and generalization ability are improved, and a certain technical guarantee is provided for the safety of the driver during driving. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is an FCNN-DPBR model structure diagram.

[0045] Figure 2 It is a definition of driving posture behavior region diagram, wherein (a) is a DPBR example in the Kaggle dataset, and (b) is a DPBR example in the SEU dataset.

[0046] Figure 3 is a VGG16-DPBR model structure diagram.

[0047] Figure 4 is an Xception-DPBR model structure diagram.

[0048] Figure 5 is a MobileNetV2 model structure diagram. DETAILED DESCRIPTION

[0049] The application will be further illustrated below in combination with the drawings and specific examples.

[0050] As shown in the drawings, Figure 1 The application provides a truck driver posture behavior recognition method based on a convolutional neural network fusion model, which comprises the following steps:

[0051] S1: A truck driver posture behavior standard image dataset is constructed, which specifically comprises:

[0052] S1-1: For the Kaggle truck driver posture behavior standard dataset, 10 driving posture behavior categories including safe driving, left hand information sending, left hand phone calling, right hand information sending, right hand phone calling, operation of the center console, drinking, body stretching back, hair arranging or makeup, and talking with passengers are included, and 1500 images of each category are randomly extracted from the original Kaggle dataset, covering male and female drivers of different ages from 20 to 60 years old, and skin colors including yellow, white, and black;

[0053] S1-2: For the SEU truck driver posture behavior standard dataset, 4 driving posture behavior categories including normal driving, steering displacement, phone calling, and eating are included, and the number of images of each category is increased to 400 through data enhancement, covering 11 male drivers of different ages from 20 to 60 years old;

[0054] S1-3: The driving posture behavior region (Driving Posture Behavior Region) in the dataset image is defined, as shown in the drawings, which aims to use as little image information as possible to describe the truck driver posture behavior, and to lay a good foundation for subsequent data labeling and target detection positioning; Figure 2

[0055] S1-4: The driving posture behavior region in the images of the two truck driver posture behavior standard datasets is labeled using LabelMe software, wherein the label name of the Kaggle dataset is: {c0, c1, c2, c3, c4, c5, c6, c7, c8, c9}, and the label name of the SEU dataset is: {Pose1, Pose2, Pose3, Pose4}; ​

[0056] S1-5: Convert the label file dataset in.json format generated by using LabelMe software into the standard VOC dataset format, which is:

[0057] — VOCdevkit

[0058] — VOC2007

[0059] — Annotations

[0060] — ImageSets

[0061] — Main

[0062] — JPEGImages

[0063] Among them, the Annotations folder stores the.xml format label file; the ImageSets folder has a Main folder under it, whose content is the partition information of the dataset; the JPEGImages folder stores the original image data;

[0064] S1-6: Before training each model, randomly extract image data and divide them into training set, validation set and test set according to the ratio of 4:1:1.

[0065] S2: Construct four models for recognizing the posture and behavior of truck drivers, which are:

[0066] S2-1: Construct a Faster R-CNN-DPBR model for recognizing the posture and behavior of truck drivers;

[0067] S2-2: Construct a SSD-DPBR model for recognizing the posture and behavior of truck drivers;

[0068] S2-3: Construct a YOLOv5-DPBR model for recognizing the posture and behavior of truck drivers;

[0069] S2-4: Construct a RetinaNet-DPBR model for recognizing the posture and behavior of truck drivers;

[0070] S2-5: Conduct comparative experiments: ① Select experimental evaluation metrics, using precision, recall, and F1-score as accuracy metrics, and the average time required to detect one image (Times, in milliseconds, rounded) as a real-time performance metric; ② During the experiment, set the corresponding hyperparameters and observe the changes in the loss function curve. If the loss function curve gradually decreases and eventually stabilizes, it indicates that the curve has converged. If it continues to decrease, the hyperparameter values ​​should be adjusted until the curve converges; ③ Apply the trained model to the test set for detection, statistically analyze the evaluation metrics of each model, and conduct a comprehensive comparison. It was found that the overall performance of the RetinaNet-DPBR model is better than the other three models, indicating that the RetinaNet-DPBR model is more suitable for the truck driver posture and behavior perception recognition task.

[0071] S3: The RetinaNet-DPBR model is used to crop and extract images from the standard image dataset of truck driver posture behavior. Specifically:

[0072] S3-1: Use the RetinaNet-DPBR model to crop and extract the driving posture behavior region in the Kaggle dataset images, thereby constructing the KaggleCrop truck driver posture behavior core dataset;

[0073] S3-2: The RetinaNet-DPBR model is used to crop and extract the driving posture behavior region in the SEU dataset images, thereby constructing the SEUCrop truck driver posture behavior core dataset;

[0074] S3-3: Based on the constructed core image dataset of truck driver posture behavior, before training each model, image data is randomly selected and divided into training set, validation set and test set in a ratio of 4:1:1.

[0075] S4: Construct a truck driver posture and behavior recognition model VGG16-DPBR based on a convolutional neural network and obtain a one-dimensional feature vector F. VThe specific method is: based on the original structure of the VGG16 model, removing the top three fully connected layers, and adding a global average pooling layer, a Dropout layer with a dropout rate of 0.5, and a fully connected layer with a softmax activation function, the network identifies 10 categories in the KaggleCrop dataset: safe driving, left hand texting, left hand calling, right hand texting, right hand calling, operating center console, drinking, body stretching back, arranging hair or makeup, and talking with passengers, and 4 categories in the SEUCrop dataset: normal driving, manipulating position, calling, and eating, thereby obtaining a truck driver posture and behavior recognition model VGG16-DPBR based on a convolutional neural network, as shown in Figure 3 After training, the top fully connected layer and the Dropout layer are deleted, and finally a one-dimensional feature vector F V .

[0076] S5: Construct a truck driver posture and behavior recognition model Xception-DPBR based on a convolutional neural network, and obtain a one-dimensional feature vector F X The specific method is: based on the original structure of the Xception model, removing the top three fully connected layers, and adding a global average pooling layer, a Dropout layer with a dropout rate of 0.5, and a fully connected layer with a softmax activation function, the network identifies 10 categories in the KaggleCrop dataset: safe driving, left hand texting, left hand calling, right hand texting, right hand calling, operating center console, drinking, body stretching back, arranging hair or makeup, and talking with passengers, and 4 categories in the SEUCrop dataset: normal driving, manipulating position, calling, and eating, thereby obtaining a truck driver posture and behavior recognition model Xception-DPBR based on a convolutional neural network, as shown in Figure 4 After training, the top fully connected layer and the Dropout layer are deleted, and finally a one-dimensional feature vector F X .

[0077] S6: Construct a truck driver posture and behavior recognition model MobileNetV2-DPBR based on a convolutional neural network, and obtain a one-dimensional feature vector F MThe specific method is: based on the original structure of the MobileNetV2 model, removing the top three fully connected layers, and adding a global average pooling layer, a Dropout layer with a dropout rate of 0.5, and a fully connected layer with a softmax activation function, the network identifies the following categories: safe driving, left hand sending messages, left hand making calls, right hand sending messages, right hand making calls, operating the center console, drinking water, stretching the body backward, arranging hair or makeup, and talking with passengers in the KaggleCrop dataset, and normal driving, steering, making calls, and eating in the SEUCrop dataset, thereby obtaining a truck driver posture and behavior recognition model MobileNetV2-DPBR based on a convolutional neural network, as shown in Figure 5 After training, the top fully connected layer and the Dropout layer are deleted, and finally a one-dimensional feature vector F M .

[0078] S7: The feature vectors F V , F X , and F M are fused based on the parallel rule, thereby constructing a convolutional neural network fusion model FCNN-DPBR. The specific method is: for the VGG16-DPBR, Xception-DPBR, and MobileNetV2-DPBR models, the top fully connected layer and the Dropout layer of the original structure are removed, and the obtained one-dimensional feature vectors F V , F X , and F M are fused based on the parallel rule to obtain a feature vector of 1x1x3840. Then, the merged one-dimensional feature vector is input, a Dropout layer is added again to prevent overfitting, and a fully connected layer with a softmax activation function outputs the prediction result. The fusion model is retrained, and finally the fusion model FCNN-DPBR is obtained.

[0079] To verify the superiority of the RetinaNet-DPBR model, comparative experiments were carried out in this case. Based on the constructed Kaggle truck driver posture behavior standard dataset and SEU truck driver posture behavior dataset, the training set, validation set and test set were divided according to the ratio of 4:1:1, and the Faster R-CNN-DPBR, SSD-DPBR, YOLOv5-DPBR and RetinaNet-DPBR models were used for the cognitive recognition experiments of the two standard datasets respectively, and the neural network was trained to obtain the detection results of the test set, as shown in Tables 5-6. It can be seen that the test results of the RetinaNet-DPBR model on the two standard datasets are better than those of the other three models, indicating that RetinaNet-DPBR is more suitable for cropping the driving posture behavior region in the dataset.

[0080] Table 5 Test results of the model on the Kaggle dataset

[0081]

[0082] Table 6 Test results of the model on the SEU dataset

[0083]

[0084] In order to verify the superiority of the fusion model (FCNN-DPBR), comparative experiments were also carried out. First, the RetinaNet-DPBR model was used to crop and extract the driving posture behavior region in the image of the two standard datasets, thereby constructing the KaggleCrop dataset and SEUCrop dataset, and the training set, validation set and test set were divided according to the ratio of 4:1:1. Four truck driver posture behavior recognition models of VGG16-DPBR, Xception-DPBR, MobileNetV2-DPBR and FCNN-DPBR were trained respectively, and the detection results of different models on the test set were compared and analyzed, as shown in Tables 7-8. It can be seen that the test results of the fusion model (FCNN-DPBR) on the KaggleCrop and SEUCrop datasets are better than those of the single network model, and more importantly, compared with the single network model, the training speed of the fusion model based on the single network is faster.

[0085] Table 7 Test results of the fusion model and the single network model on the KaggleCrop dataset

[0086]

[0087] Table 8 Test results of the fusion model and the single network model on the SEUCrop dataset

[0088]

[0089]

[0090] In order to further show the effectiveness of the FCNN-DPBR model, the confusion matrix of the test set recognition result is counted, as shown in Tables 9-10. The results show that the FCNN-DPBR fusion model has good recognition performance for truck driver posture behavior, and the recognition accuracy of various driving postures is not less than 98%.

[0091] The technical means disclosed in the present application scheme is not limited to the technical means disclosed in the above-mentioned embodiments, but also includes the technical solutions composed of any combination of the above technical features. It should be pointed out that for ordinary skilled persons in the art, without departing from the principle of the present application, some improvements and refinements can be made, and these improvements and refinements are also considered as the protection scope of the present application.

[0092] Table 9 Confusion matrix of FCNN-DPBR fusion model on SEUCrop dataset

[0093] Pose 1 Pose 2 Pose 3 Pose 4 Pose 1 100％ 0 0 0 Pose 2 0 98.50％ 1.50％ 0 Pose 3 0 0 100％ Pose 4 0 0 0 100％

[0094] Table 10 Confusion matrix of FCNN-DPBR fusion model on KaggleCrop dataset

[0095]

[0096]

Claims

1. A truck driver posture behavior recognition method based on a convolutional neural network fusion model, characterized in that: Comprise the following steps: S1: build truck driver posture behavior standard image dataset, Kaggle open source dataset and SEU self-made dataset; S2: build four models for truck driver posture behavior recognition: Faster R-CNN-DPBR, SSD-DPBR, YOLOv5-DPBR, RetinaNet-DPBR, and based on the two standard image datasets and comparative test, the most suitable model for truck driver posture behavior recognition task and the best comprehensive performance model is RetinaNet-DPBR; S3: use RetinaNet-DPBR model to crop and extract truck driver posture behavior standard image dataset, get the driving posture behavior region in the image, and build truck driver posture behavior core image dataset, KaggleCrop dataset and SEUCrop dataset; S4: Construct a truck driver posture behavior recognition model VGG16-DPBR based on a convolutional neural network, and obtain a one-dimensional feature vector F V ; S5: Construct a truck driver posture behavior recognition model Xception-DPBR based on a convolutional neural network, and obtain a one-dimensional feature vector F X ; S6: Construct a truck driver posture behavior recognition model MobileNetV2-DPBR based on a convolutional neural network to obtain a one-dimensional feature vector F M ; S7: the feature vector F V , F X , F M Based on the parallel rule, the convolutional neural network fusion model FCNN-DPBR is constructed, and the posture behavior of truck drivers is identified. The specific method for building the truck driver posture behavior standard image dataset in the step S1 is as follows: S1-1: for Kaggle truck driver posture behavior standard dataset, 10 driving posture behavior categories including safe driving, left hand sending message, left hand making phone call, right hand sending message, right hand making phone call, operating center console, drinking water, body stretching back, arranging hair or making up, and talking with passenger are included, 1500 images of each category are randomly extracted from the original Kaggle dataset, and male and female drivers of different ages from 20 to 60 years old are covered, and skin colors include yellow, white and black; S1-2: for SEU truck driver posture behavior standard dataset, 4 driving posture behavior categories including normal driving, steering displacement, making phone call and eating are included, the number of images of each category is increased to 400 through data enhancement, and 11 male drivers of different ages from 20 to 60 years old are covered; S1-3: define the driving posture behavior region in the dataset image, the purpose is to use as little image information as possible to describe the truck driver posture behavior, and lay a good foundation for subsequent data labeling and target detection positioning; S1-4: use LabelMe software to label the driving posture behavior region in the two truck driver posture behavior standard dataset images, wherein the label name of Kaggle dataset is: {c0, c1, c2, c3, c4, c5, c6, c7, c8, c9}, and the label name of SEU dataset is: {Pose1, Pose2, Pose3, Pose4}; S1-5: convert the.json format label file dataset generated by using LabelMe software into standard VOC dataset format, which is: VOCdevkit VOC2007 Annotations ImageSets Main JPEGImages Among them, the Annotations folder stores the.xml format tag file; The ImageSets folder has a Main folder under it, which contains the partition information of the dataset; The JPEGImages folder stores the original image data; S1-6: Before training each model, randomly extract image data and divide it into training set, validation set and test set according to the ratio of 4:1:

1.

2. The truck driver posture behavior recognition method based on the convolutional neural network fusion model according to claim 1, characterized in that: The specific steps of constructing four models for identifying the posture behavior of truck drivers in step S2 are as follows: S2-1: Construct a Faster R-CNN-DPBR model for identifying the posture behavior of truck drivers, and the model structure parameters are shown in Table 1, where k represents the image resolution in Kaggle dataset, which is 640x480; s represents the image resolution in SEU dataset, which is 360x480; Table 1 Faster R-CNN-DPBR model structure parameters ； S2-2: Construct a SSD-DPBR model for identifying the posture behavior of truck drivers, and the model structure parameters are shown in Table 2; Table 2 SSD-DPBR model structure parameters ； S2-3: Construct a YOLOv5-DPBR model for identifying the posture behavior of truck drivers, and the model structure parameters are shown in Table 3; Table 3 YOLOv5-DPBR model structure parameters ； S2-4: Construct a RetinaNet-DPBR model for identifying the posture behavior of truck drivers, and the model structure parameters are shown in Table 4; Table 4 RetinaNet-DPBR model structure parameters ； S2-5: Conduct comparative experiments to compare the test results of the comparative models on Kaggle dataset and SEU dataset, and obtain the model with the best comprehensive performance in the two datasets.

3. The truck driver posture and behavior recognition method based on the convolutional neural network fusion model according to claim 2, characterized in that: The specific content of the comparative experiment in step S2-5 is as follows: ① Select experimental evaluation indicators, use precision, recall and F1-score as accuracy indicators, and average detection time per image as real-time indicator; ② In the experimental stage, the corresponding hyperparameters need to be set, and the change of the loss function curve is observed. If the loss function curve gradually decreases and finally remains stable, it indicates that the curve has converged. If it is always in a downward trend, the value of the hyperparameter should be adjusted until the curve converges; ③ Apply the trained model to the test set for detection, calculate the evaluation indicators of each model, and compare them comprehensively. It is found that the comprehensive performance of RetinaNet-DPBR model is better than that of the other three models, indicating that RetinaNet-DPBR model is more suitable for identifying the posture behavior of truck drivers.

4. The truck driver posture and behavior recognition method based on the convolutional neural network fusion model according to claim 1, characterized in that: The specific content of using RetinaNet-DPBR model to crop and extract truck driver posture behavior standard image dataset in step S3 is as follows: S3-1: Crop and extract the driving posture behavior region in the Kaggle dataset image using the RetinaNet-DPBR model, thereby constructing the KaggleCrop truck driver posture behavior core dataset; S3-2: Crop and extract the driving posture behavior region in the SEU dataset image using the RetinaNet-DPBR model, thereby constructing the SEUCrop truck driver posture behavior core dataset; S3-3: Based on the constructed truck driver posture behavior core image dataset, randomly extract image data before training each model, and divide the training set, validation set and test set according to the ratio of 4:1:

1.

5. The truck driver posture and behavior recognition method based on the convolutional neural network fusion model according to claim 1, characterized in that: The step S4 constructs a truck driver posture behavior recognition model VGG16-DPBR based on a convolutional neural network, and obtains a one-dimensional feature vector F V The specific method is: based on the original structure of the VGG16 model, removing the top three fully connected layers, and adding a global average pooling layer, a Dropout layer with a dropout rate of 0.5, and a fully connected layer with a softmax activation function on this basis. The network recognition categories are: safe driving, left hand texting, left hand calling, right hand texting, right hand calling, operating center console, drinking, body stretching back, arranging hair or makeup, and talking with passengers in the KaggleCrop dataset, and normal driving, steering, calling, and eating in the SEUCrop dataset, thereby obtaining a truck driver posture behavior recognition model VGG16-DPBR based on a convolutional neural network. After training, the top fully connected layer and the Dropout layer are deleted, and finally a one-dimensional feature vector F V of 1×1×512 is output.

6. The truck driver posture and behavior recognition method based on the convolutional neural network fusion model according to claim 1, characterized in that: The step S5 constructs a truck driver posture behavior recognition model Xception-DPBR based on a convolutional neural network, and obtains a one-dimensional feature vector F X The specific method is: based on the original structure of the Xception model, removing the top three fully connected layers, and adding a global average pooling layer, a Dropout layer with a dropout rate of 0.5, and a fully connected layer with a softmax activation function. The network recognition categories are: safe driving, left hand texting, left hand calling, right hand texting, right hand calling, operating center console, drinking, body stretching back, arranging hair or makeup, and talking with passengers in the KaggleCrop dataset; normal driving, steering, calling, and eating in the SEUCrop dataset; thereby obtaining a truck driver posture behavior recognition model Xception-DPBR based on a convolutional neural network. After training, the top fully connected layer and the Dropout layer are deleted, and finally a one-dimensional feature vector F X of 1×1×2048 is output.

7. The truck driver posture and behavior recognition method based on the convolutional neural network fusion model according to claim 1, characterized in that: The step S6 constructs a truck driver posture behavior recognition model MobileNetV2-DPBR based on a convolutional neural network, and obtains a one-dimensional feature vector F M The specific method is: based on the original structure of the MobileNetV2 model, removing the top three fully connected layers, and adding a global average pooling layer, a Dropout layer with a dropout rate of 0.5, and a fully connected layer with a softmax activation function on this basis. The network recognition categories are: safe driving, left hand sending messages, left hand making phone calls, right hand sending messages, right hand making phone calls, operating the center console, drinking water, stretching the body backward, arranging hair or makeup, and talking with passengers in the KaggleCrop dataset, and normal driving, steering displacement, making phone calls, and eating in the SEUCrop dataset. Thus, a truck driver posture behavior recognition model MobileNetV2-DPBR based on a convolutional neural network is obtained. After training, the top fully connected layer and the Dropout layer are deleted, and finally a one-dimensional feature vector F M of 1×1×1280 is output. 8.The truck driver posture behavior recognition method based on the convolutional neural network fusion model according to claim 1, characterized in that: The feature vector F V , F X , F M Based on the parallel rule, the specific method for constructing the convolutional neural network fusion model FCNN-DPBR is: extracting the feature vectors F V , F X , F M The parallel fusion rule is adopted to form the FCNN-DPBR wagon driver posture behavior recognition model. 9.The truck driver posture behavior recognition method based on the convolutional neural network fusion model according to claim 8, characterized in that: The specific content of the parallel fusion rule is that for the three models of VGG16-DPBR, Xception-DPBR and MobileNetV2-DPBR, the fully connected layer and the Dropout layer at the top of the original structure are removed, and the obtained one-dimensional feature vector F V , F X , F M Parallel fusion obtains a feature of 1×1×3840, then the merged one-dimensional feature vector is taken as an input, a Dropout layer is added again to prevent overfitting, and a fully connected layer with a softmax activation function outputs a prediction result, the fully connected layer of the fusion model is retrained, and finally the fusion model FCNN-DPBR is obtained.

Images